This is a multi-part message in MIME format. --------------C563F370DBD059BBFD94E893 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/20/2017 02:20 PM, yayo (j) wrote:
Hi,
Thank you for the answer and sorry for delay:
2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishankar@redhat.com <mailto:ravishankar@redhat.com>>:
1. What does the glustershd.log say on all 3 nodes when you run the command? Does it complain anything about these files?
No, glustershd.log is clean, no extra log after command on all 3 nodes
Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that. If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen. If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex //gluster/engine/brick//path-to-file` ? Thanks, Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this volume?
/Volume Name: engine/ /Type: Replicate/ /Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/ /Status: Started/ /Snapshot Count: 0/ /Number of Bricks: 1 x 3 = 3/ /Transport-type: tcp/ /Bricks:/ /Brick1: node01:/gluster/engine/brick/ /Brick2: node02:/gluster/engine/brick/ /Brick3: node04:/gluster/engine/brick/ /Options Reconfigured:/ /nfs.disable: on/ /performance.readdir-ahead: on/ /transport.address-family: inet/ /storage.owner-uid: 36/ /performance.quick-read: off/ /performance.read-ahead: off/ /performance.io-cache: off/ /performance.stat-prefetch: off/ /performance.low-prio-threads: 32/ /network.remote-dio: off/ /cluster.eager-lock: enable/ /cluster.quorum-type: auto/ /cluster.server-quorum-type: server/ /cluster.data-self-heal-algorithm: full/ /cluster.locking-scheme: granular/ /cluster.shd-max-threads: 8/ /cluster.shd-wait-qlength: 10000/ /features.shard: on/ /user.cifs: off/ /storage.owner-gid: 36/ /features.shard-block-size: 512MB/ /network.ping-timeout: 30/ /performance.strict-o-direct: on/ /cluster.granular-entry-heal: on/ /auth.allow: */
server.allow-insecure: on
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node02:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
Thanks, Ravi
Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out
Thank you!
--------------C563F370DBD059BBFD94E893 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 07/20/2017 02:20 PM, yayo (j) wrote:<br> </div> <blockquote type="cite" cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">Hi, </div> <div class="gmail_quote"><br> </div> <div class="gmail_quote">Thank you for the answer and sorry for delay:</div> <div class="gmail_quote"><br> </div> <div class="gmail_quote">2017-07-19 16:55 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>></span>:</div> <div class="gmail_quote"><br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 1. What does the glustershd.log say on all 3 nodes when you run the command? Does it complain anything about these files?<br> </div> </blockquote> <div><br> </div> <div>No, glustershd.log is clean, no extra log after command on all 3 nodes</div> </div> </div> </div> </blockquote> <br> Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that.<br> If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.<br> <br> If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex <i>/gluster/engine/brick/</i>path-to-file` ?<br> <br> Thanks,<br> Ravi<br> <br> <blockquote type="cite" cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 2. Are these 12 files also present in the 3rd data brick?<br> </div> </blockquote> <div><br> </div> <div>I've checked right now: all files exists in all 3 nodes </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 3. Can you provide the output of `gluster volume info` for the this volume?</div> </blockquote> <div><br> </div> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume Name: engine</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Type: Replicate</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Status: Started</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Snapshot Count: 0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Bricks: 1 x 3 = 3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Transport-type: tcp</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Bricks:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick1: node01:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick2: node02:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick3: node04:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Options Reconfigured:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>nfs.disable: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.readdir-ahead: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>transport.address-family: inet</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-uid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.quick-read: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.read-ahead: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.io-cache: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.stat-prefetch: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.low-prio-threads: 32</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.remote-dio: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.eager-lock: enable</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.quorum-type: auto</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.server-quorum-type: server</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.data-self-heal-algorithm: full</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.locking-scheme: granular</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-max-threads: 8</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-wait-qlength: 10000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>user.cifs: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-gid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard-block-size: 512MB</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.ping-timeout: 30</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.strict-o-direct: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.granular-entry-heal: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>auth.allow: *</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div> server.allow-insecure: on</div> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="gmail-"><br> <blockquote type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div>Some extra info:</div> <div><br> </div> <div>We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster </div> </div> </blockquote> </div> </div> </blockquote> <br> </span> Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?<br> <br> </div> </blockquote> <div><br> </div> <div>Yes</div> <br> <div> <div class="gmail-hljs gmail-nginx" style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace"> <div><br> </div> <div> <div><span class="gmail-hljs-attribute">#gluster</span> volume remove-brick engine replica <span class="gmail-hljs-number" style="color:rgb(136,0,0)">2</span> node03:/gluster/data/brick force <b>(OK!)</b></div> <div><br> </div> <div>#gluster volume heal engine <span class="gmail-hljs-literal" style="color:rgb(120,169,96)">info</span> <b>(<span class="gmail-hljs-literal" style="color:rgb(120,169,96)">no</span> entries!)</b></div> <div><br> </div> <div>#gluster volume add-brick engine replica <span class="gmail-hljs-number" style="color:rgb(136,0,0)">3</span> node04:/gluster/engine/brick <b>(OK!)</b></div> <div><br> </div> <div><b>After some minutes</b></div> <div><br> </div> <div>[root<span class="gmail-hljs-variable" style="color:rgb(188,96,96)">@node01</span> ~]<span class="gmail-hljs-comment" style="color:rgb(136,136,136)"># gluster volume heal engine info</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node01:/gluster/engine/brick</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node02:/gluster/engine/brick</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node04:/gluster/engine/brick</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> </div> </div> </div> <br> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> Thanks,<br> Ravi</div> </blockquote> <div><br> </div> <div>Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out </div> <div><br> </div> <div>Thank you!</div> </div> <div><br> </div> </div> </div> </blockquote> <br> </body> </html> --------------C563F370DBD059BBFD94E893--