This is a multi-part message in MIME format. --------------5903BBB64769515A20658214 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/20/2017 03:42 PM, yayo (j) wrote:
2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishankar@redhat.com <mailto:ravishankar@redhat.com>>:
Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that. If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.
I've executed the command on all 3 nodes (Know is enougth only one) , after that the "heal" command report elements between 6 and 10 ... (sometime 6, sometime 8, sometime 10)
Log on glustershd.log don't say anything :
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
/[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/ /[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81/ /[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex //gluster/engine/brick//path-to-file` ?
*/NODE01:/* /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000120000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000000e0000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x676067891f344c1586b8c0d05b07f187/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000550000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000070000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000000000000000000000/ /trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9/ /trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/__DIRECT_IO_TEST__/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000000000000000000000/ /trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000010000000000000000/ /trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9/ /trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000000a0000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/ / / */ /* */NODE02:/* /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000001a0000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/ /trusted.afr.dirty=0x000000010000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000000c0000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x676067891f344c1586b8c0d05b07f187/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000008e0000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000090000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000010000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/__DIRECT_IO_TEST__/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000000000000000000000/ /trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000020000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000120000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/ / / / / / / / / /*NODE04*:/ /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x676067891f344c1586b8c0d05b07f187/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/__DIRECT_IO_TEST__/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x0200000000000000596484e20006237b/ /trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/
hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how?
/[root@node01 ~]# sestatus/ /SELinux status: disabled/ / / /[root@node02 ~]# sestatus/ /SELinux status: disabled/ / / /[root@node04 ~]# sestatus/ /SELinux status: disabled/
Thank you
Thanks, Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this volume?
/Volume Name: engine/ /Type: Replicate/ /Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/ /Status: Started/ /Snapshot Count: 0/ /Number of Bricks: 1 x 3 = 3/ /Transport-type: tcp/ /Bricks:/ /Brick1: node01:/gluster/engine/brick/ /Brick2: node02:/gluster/engine/brick/ /Brick3: node04:/gluster/engine/brick/ /Options Reconfigured:/ /nfs.disable: on/ /performance.readdir-ahead: on/ /transport.address-family: inet/ /storage.owner-uid: 36/ /performance.quick-read: off/ /performance.read-ahead: off/ /performance.io-cache: off/ /performance.stat-prefetch: off/ /performance.low-prio-threads: 32/ /network.remote-dio: off/ /cluster.eager-lock: enable/ /cluster.quorum-type: auto/ /cluster.server-quorum-type: server/ /cluster.data-self-heal-algorithm: full/ /cluster.locking-scheme: granular/ /cluster.shd-max-threads: 8/ /cluster.shd-wait-qlength: 10000/ /features.shard: on/ /user.cifs: off/ /storage.owner-gid: 36/ /features.shard-block-size: 512MB/ /network.ping-timeout: 30/ /performance.strict-o-direct: on/ /cluster.granular-entry-heal: on/ /auth.allow: */
server.allow-insecure: on
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node02:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
Thanks, Ravi
Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out
Thank you!
-- Linux User: 369739 http://counter.li.org
--------------5903BBB64769515A20658214 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p><br> </p> <br> <div class="moz-cite-prefix">On 07/20/2017 03:42 PM, yayo (j) wrote:<br> </div> <blockquote type="cite" cite="mid:CAGK=3kycgTSLxRLZSPNZi+ERh_=OhDFH29hXHzWEPnsV3vmhAQ@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"><br> <div class="gmail_quote">2017-07-20 11:34 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="gmail-m_7883186325552308780gmail-"> <br> <div class="gmail-m_7883186325552308780gmail-m_9132661499198415129moz-cite-prefix">Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that.<br> </div> </span> If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.<br> <br> </div> </blockquote> <div><br> </div> <div>I've executed the command on all 3 nodes (Know is enougth only one) , after that the "heal" command report elements between 6 and 10 ... (sometime 6, sometime 8, sometime 10)</div> <div><br> </div> <div><br> </div> <div>Log on glustershd.log don't say anything :</div> </div> </div> </div> </blockquote> <br> But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case.<br> Hope this helps,<br> Ravi<br> <br> <blockquote type="cite" cite="mid:CAGK=3kycgTSLxRLZSPNZi+ERh_=OhDFH29hXHzWEPnsV3vmhAQ@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:<wbr>afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-<wbr>7b6f5bd74327. sources=[0] 1 sinks=2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-<wbr>5b6974bcef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:<wbr>afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-<wbr>5b6974bcef81. sources=[0] 1 sinks=2</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex <i>/gluster/engine/brick/</i>path-to-<wbr>file` ?<br> <br> </div> </blockquote> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><b><i>NODE01:</i></b></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000120000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000000e0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000550000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000070000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x0f0000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000010000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x0f0000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000000a0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><b><i><br> </i></b></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><b><i>NODE02:</i></b></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000001a0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000010000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000000c0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000008e0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000090000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000010000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000020000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000120000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><b>NODE04</b>:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x0200000000000000596484e20006<wbr>237b</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div>hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how?</div> <div><br> </div> </div> </div> <blockquote style="margin:0 0 0 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[root@node01 ~]# sestatus</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>SELinux status: disabled</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[root@node02 ~]# sestatus</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>SELinux status: disabled</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[root@node04 ~]# sestatus</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>SELinux status: disabled</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div><br> </div> <div>Thank you </div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> Thanks,<br> Ravi <div> <div class="gmail-m_7883186325552308780gmail-h5"><br> <br> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 2. Are these 12 files also present in the 3rd data brick?<br> </div> </blockquote> <div><br> </div> <div>I've checked right now: all files exists in all 3 nodes </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 3. Can you provide the output of `gluster volume info` for the this volume?</div> </blockquote> <div><br> </div> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume Name: engine</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Type: Replicate</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume ID: d19c19e3-910d-437b-8ba7-4f2a23<wbr>d17515</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Status: Started</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Snapshot Count: 0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Bricks: 1 x 3 = 3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Transport-type: tcp</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Bricks:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick1: node01:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick2: node02:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick3: node04:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Options Reconfigured:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>nfs.disable: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.readdir-ahead: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>transport.address-family: inet</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-uid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.quick-read: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.read-ahead: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.io-cache: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.stat-prefetch: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.low-prio-threads: 32</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.remote-dio: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.eager-lock: enable</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.quorum-type: auto</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.server-quorum-type: server</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.data-self-heal-algorit<wbr>hm: full</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.locking-scheme: granular</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-max-threads: 8</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-wait-qlength: 10000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>user.cifs: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-gid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard-block-size: 512MB</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.ping-timeout: 30</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.strict-o-direct: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.granular-entry-heal: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>auth.allow: *</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div> server.allow-insecure: on</div> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-"><br> <blockquote type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div>Some extra info:</div> <div><br> </div> <div>We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster </div> </div> </blockquote> </div> </div> </blockquote> <br> </span> Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?<br> <br> </div> </blockquote> <div><br> </div> <div>Yes</div> <br> <div> <div class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-nginx" style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace"> <div><br> </div> <div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-attribute">#gluster</span> volume remove-brick engine replica <span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-number" style="color:rgb(136,0,0)">2</span> node03:/gluster/data/brick force <b>(OK!)</b></div> <div><br> </div> <div>#gluster volume heal engine <span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-literal" style="color:rgb(120,169,96)">info</span> <b>(<span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-literal" style="color:rgb(120,169,96)">no</span> entries!)</b></div> <div><br> </div> <div>#gluster volume add-brick engine replica <span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-number" style="color:rgb(136,0,0)">3</span> node04:/gluster/engine/brick <b>(OK!)</b></div> <div><br> </div> <div><b>After some minutes</b></div> <div><br> </div> <div>[root<span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-variable" style="color:rgb(188,96,96)">@node01</span> ~]<span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)"># gluster volume heal engine info</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node01:/gluster/engine/brick</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node02:/gluster/engine/brick</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node04:/gluster/engine/brick</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> </div> </div> </div> <br> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> Thanks,<br> Ravi</div> </blockquote> <div><br> </div> <div>Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out </div> <div><br> </div> <div>Thank you!</div> </div> <div><br> </div> </div> </div> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> <br clear="all"> <div><br> </div> -- <br> <div class="gmail-m_7883186325552308780gmail_signature">Linux User: 369739 <a href="http://counter.li.org" target="_blank" moz-do-not-send="true">http://counter.li.org</a></div> </div> </div> </blockquote> <br> </body> </html> --------------5903BBB64769515A20658214--