4:34 a.m.
This is a multi-part message in MIME format.
--------------C563F370DBD059BBFD94E893
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 07/20/2017 02:20 PM, yayo (j) wrote:
Hi,
Thank you for the answer and sorry for delay:
2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com
<mailto:ravishankar@redhat.com>>:
1. What does the glustershd.log say on all 3 nodes when you run
the command? Does it complain anything about these files?
No, glustershd.log is clean, no extra log after command on all 3 nodes
Could you check if the self-heal daemon on all nodes is connected to the
3 bricks? You will need to check the glustershd.log for that.
If it is not connected, try restarting the shd using `gluster volume
start engine force`, then launch the heal command like you did earlier
and see if heals happen.
If it doesn't, please provide the getfattr outputs of the 12 files from
all 3 nodes using `getfattr -d -m . -e hex
//gluster/engine/brick//path-to-file` ?
Thanks,
Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the
this volume?
/Volume Name: engine/
/Type: Replicate/
/Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: node01:/gluster/engine/brick/
/Brick2: node02:/gluster/engine/brick/
/Brick3: node04:/gluster/engine/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: off/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 10000/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */
server.allow-insecure: on
> Some extra info:
>
> We have recently changed the gluster from: 2 (full
> repliacated) + 1 arbiter to 3 full replicated cluster
>
Just curious, how did you do this? `remove-brick` of arbiter
brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2
node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3
node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node02:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
Thanks,
Ravi
Another extra info (I don't know if this can be the problem): Five
days ago A black out has suddenly shut down the networks switch (also
gluster network) of node 03 and 04 ... But I don't know this problem
is in place after this black out
Thank you!
--------------C563F370DBD059BBFD94E893
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/20/2017 02:20 PM, yayo (j)
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Hi, </div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Thank you for the answer and sorry
for delay:</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">2017-07-19 16:55 GMT+02:00
Ravishankar N <span dir="ltr"><<a
href="mailto:ravishankar@redhat.com" target="_blank"
moz-do-not-send="true">ravishankar@redhat.com</a>></span>:</div>
<div class="gmail_quote"><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 1. What does the glustershd.log
say on all 3 nodes when you run the command? Does it
complain anything about these files?<br>
</div>
</blockquote>
<div><br>
</div>
<div>No, glustershd.log is clean, no extra log after command
on all 3 nodes</div>
</div>
</div>
</div>
</blockquote>
<br>
Could you check if the self-heal daemon on all nodes is connected to
the 3 bricks? You will need to check the glustershd.log for that.<br>
If it is not connected, try restarting the shd using `gluster volume
start engine force`, then launch the heal command like you did
earlier and see if heals happen.<br>
<br>
If it doesn't, please provide the getfattr outputs of the 12 files
from all 3 nodes using `getfattr -d -m . -e hex
<i>/gluster/engine/brick/</i>path-to-file`
?<br>
<br>
Thanks,<br>
Ravi<br>
<br>
<blockquote type="cite"
cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 2. Are these 12 files also present
in the 3rd data brick?<br>
</div>
</blockquote>
<div><br>
</div>
<div>I've checked right now: all files exists in all 3
nodes </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 3. Can you provide the output of
`gluster volume info` for the this volume?</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume Name: engine</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Type: Replicate</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume ID:
d19c19e3-910d-437b-8ba7-4f2a23d17515</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Status: Started</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Snapshot Count: 0</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Bricks: 1 x 3 = 3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Transport-type: tcp</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Bricks:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick1:
node01:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick2:
node02:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick3:
node04:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Options Reconfigured:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>nfs.disable: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.readdir-ahead: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>transport.address-family: inet</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-uid: 36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.quick-read: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.read-ahead: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.io-cache: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.stat-prefetch: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.low-prio-threads:
32</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.remote-dio: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.eager-lock: enable</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.quorum-type: auto</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.server-quorum-type:
server</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.data-self-heal-algorithm:
full</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.locking-scheme:
granular</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-max-threads: 8</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-wait-qlength: 10000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>user.cifs: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-gid: 36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard-block-size:
512MB</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.ping-timeout: 30</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.strict-o-direct: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.granular-entry-heal: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>auth.allow: *</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div> server.allow-insecure: on</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span
class="gmail-"><br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Some extra info:</div>
<div><br>
</div>
<div>We have recently changed the gluster
from: 2 (full repliacated) + 1 arbiter to
3 full replicated cluster </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</span> Just curious, how did you do this?
`remove-brick` of arbiter brick followed by an
`add-brick` to increase to replica-3?<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes</div>
<br>
<div>
<div class="gmail-hljs gmail-nginx"
style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace">
<div><br>
</div>
<div>
<div><span
class="gmail-hljs-attribute">#gluster</span>
volume remove-brick engine replica <span
class="gmail-hljs-number"
style="color:rgb(136,0,0)">2</span>
node03:/gluster/data/brick force <b>(OK!)</b></div>
<div><br>
</div>
<div>#gluster volume heal engine <span
class="gmail-hljs-literal"
style="color:rgb(120,169,96)">info</span>
<b>(<span
class="gmail-hljs-literal"
style="color:rgb(120,169,96)">no</span>
entries!)</b></div>
<div><br>
</div>
<div>#gluster volume add-brick engine replica <span
class="gmail-hljs-number"
style="color:rgb(136,0,0)">3</span>
node04:/gluster/engine/brick <b>(OK!)</b></div>
<div><br>
</div>
<div><b>After some minutes</b></div>
<div><br>
</div>
<div>[root<span class="gmail-hljs-variable"
style="color:rgb(188,96,96)">@node01</span>
~]<span
class="gmail-hljs-comment"
style="color:rgb(136,136,136)"># gluster volume
heal engine info</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node01:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
<div><br>
</div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node02:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
<div><br>
</div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node04:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
</div>
</div>
</div>
<br>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> Thanks,<br>
Ravi</div>
</blockquote>
<div><br>
</div>
<div>Another extra info (I don't know if this can be the
problem): Five days ago A black out has suddenly shut down
the networks switch (also gluster network) of node 03 and
04 ... But I don't know this problem is in place after
this black out </div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------C563F370DBD059BBFD94E893--