ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
4:22 a.m.
Hi all,
We have an ovirt cluster hyperconverged with hosted engine on 3 full
replicated node . This cluster have 2 gluster volume:
- data: volume for the Data (Master) Domain (For vm)
- engine: volume fro the hosted_storage Domain (for hosted engine)
We have this problem: "engine" gluster volume have always unsynced elements
and we cant' fix the problem, on command line we have tried to use the
"heal" command but elements remain always unsynced ....
Below the heal command "status":
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 12
Brick node02:/gluster/engine/brick
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
<gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f>
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
<gfid:1e309376-c62e-424f-9857-f9a0c3a729bf>
<gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9>
<gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a>
/__DIRECT_IO_TEST__
<gfid:67606789-1f34-4c15-86b8-c0d05b07f187>
<gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0>
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
<gfid:9ad720b2-507d-4830-8294-ec8adee6d384>
<gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf>
Status: Connected
Number of entries: 12
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
running the "gluster volume heal engine" don't solve the problem...
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter
to 3 full replicated cluster but i don't know this is the problem...
The "data" volume is good and healty and have no unsynced entry.
Ovirt refuse to put the node02 and node01 in "maintenance mode" and
complains about "unsynced elements"
How can I fix this?
Thank you
9:32 a.m.
[Adding gluster-users]
On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) <jaganz(a)gmail.com> wrote:
Hi all,
We have an ovirt cluster hyperconverged with hosted engine on 3 full
replicated node . This cluster have 2 gluster volume:
- data: volume for the Data (Master) Domain (For vm)
- engine: volume fro the hosted_storage Domain (for hosted engine)
We have this problem: "engine" gluster volume have always unsynced
elements and we cant' fix the problem, on command line we have tried to use
the "heal" command but elements remain always unsynced ....
Below the heal command "status":
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-
52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-
a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 12
Brick node02:/gluster/engine/brick
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-
52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
<gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f>
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
<gfid:1e309376-c62e-424f-9857-f9a0c3a729bf>
<gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9>
<gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a>
/__DIRECT_IO_TEST__
<gfid:67606789-1f34-4c15-86b8-c0d05b07f187>
<gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0>
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-
a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
<gfid:9ad720b2-507d-4830-8294-ec8adee6d384>
<gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf>
Status: Connected
Number of entries: 12
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
running the "gluster volume heal engine" don't solve the problem...
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1
arbiter to 3 full replicated cluster but i don't know this is the problem...
The "data" volume is good and healty and have no unsynced entry.
Ovirt refuse to put the node02 and node01 in "maintenance mode" and
complains about "unsynced elements"
How can I fix this?
Thank you
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:55 a.m.
This is a multi-part message in MIME format.
--------------0C293E7D7C0723BAFB7F6D00
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 07/19/2017 08:02 PM, Sahina Bose wrote:
[Adding gluster-users]
On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) <jaganz(a)gmail.com
<mailto:jaganz@gmail.com>> wrote:
Hi all,
We have an ovirt cluster hyperconverged with hosted engine on 3
full replicated node . This cluster have 2 gluster volume:
- data: volume for the Data (Master) Domain (For vm)
- engine: volume fro the hosted_storage Domain (for hosted engine)
We have this problem: "engine" gluster volume have always unsynced
elements and we cant' fix the problem, on command line we have
tried to use the "heal" command but elements remain always
unsynced ....
Below the heal command "status":
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 12
Brick node02:/gluster/engine/brick
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
<gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f>
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
<gfid:1e309376-c62e-424f-9857-f9a0c3a729bf>
<gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9>
<gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a>
/__DIRECT_IO_TEST__
<gfid:67606789-1f34-4c15-86b8-c0d05b07f187>
<gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0>
/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
<gfid:9ad720b2-507d-4830-8294-ec8adee6d384>
<gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf>
Status: Connected
Number of entries: 12
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
running the "gluster volume heal engine" don't solve the problem...
1. What does the glustershd.log say on all 3 nodes when you run the
command? Does it complain anything about these files?
2. Are these 12 files also present in the 3rd data brick?
3. Can you provide the output of `gluster volume info` for the this volume?
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) +
1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick
followed by an `add-brick` to increase to replica-3?
Thanks,
Ravi
but i don't know this is the problem...
The "data" volume is good and healty and have no unsynced entry.
Ovirt refuse to put the node02 and node01 in "maintenance mode"
and complains about "unsynced elements"
How can I fix this?
Thank you
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
--------------0C293E7D7C0723BAFB7F6D00
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 07/19/2017 08:02 PM, Sahina Bose
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com">
<div dir="ltr">[Adding gluster-users]<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Jul 19, 2017 at 2:52 PM, yayo
(j) <span dir="ltr"><<a
href="mailto:jaganz@gmail.com"
target="_blank"
moz-do-not-send="true">jaganz(a)gmail.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>We have an ovirt cluster hyperconverged with hosted
engine on 3 full replicated node . This cluster have 2
gluster volume:</div>
<div><br>
</div>
<div>- data: volume for the Data (Master) Domain (For vm)</div>
<div>- engine: volume fro the hosted_storage Domain (for
hosted engine)</div>
<div><br>
</div>
<div>We have this problem: "engine" gluster volume have
always unsynced elements and we cant' fix the problem,
on command line we have tried to use the "heal" command
but elements remain always unsynced .... </div>
<div><br>
</div>
<div>Below the heal command "status":</div>
<div><br>
</div>
<div>
<div class="m_6699510126617823271gmail-hljs
m_6699510126617823271gmail-objectivec"
style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace">
<div>[root@node01 ~]<span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"># gluster volume
heal engine info</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Brick
node01:/gluster/engine/brick</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.48</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.64</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.60</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.2</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.68</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/19d71267-<wbr>52a4-42a3-bb1e-e3145361c0c2/<wbr>7a215635-02f3-47db-80db-<wbr>8b689c6a8f01</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/88d41053-<wbr>a257-4272-9e2e-2f3de0743b81/<wbr>6573ed08-d3ed-4d12-9227-<wbr>2c95941e1ad6</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.61</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.1</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/dom_md/ids</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.20</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/__DIRECT_IO_TEST__</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Status:
Connected</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Number of entries:
12</span></div>
<div><br>
</div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Brick
node02:/gluster/engine/brick</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/19d71267-<wbr>52a4-42a3-bb1e-e3145361c0c2/<wbr>7a215635-02f3-47db-80db-<wbr>8b689c6a8f01</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:9a601373-bbaa-44d8-b396-<wbr>f0b9b12c026f></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/dom_md/ids</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:1e309376-c62e-424f-9857-<wbr>f9a0c3a729bf></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:e3565b50-1495-4e5b-ae88-<wbr>3bceca47b7d9></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:4e33ac33-dddb-4e29-b4a3-<wbr>51770b81166a></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/__DIRECT_IO_TEST__</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:67606789-1f34-4c15-86b8-<wbr>c0d05b07f187></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:9ef88647-cfe6-4a35-a38c-<wbr>a5173c9e8fc0></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/88d41053-<wbr>a257-4272-9e2e-2f3de0743b81/<wbr>6573ed08-d3ed-4d12-9227-<wbr>2c95941e1ad6</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:9ad720b2-507d-4830-8294-<wbr>ec8adee6d384></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)"><span
class="m_6699510126617823271gmail-hljs-meta-string"
style="color:rgb(77,153,191)"><gfid:d9853e5d-a2bf-4cee-8b39-<wbr>7781a98033cf></span></span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Status:
Connected</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Number of entries:
12</span></div>
<div><br>
</div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Brick
node04:/gluster/engine/brick</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Status:
Connected</span></div>
<div><span
class="m_6699510126617823271gmail-hljs-meta"
style="color:rgb(31,113,153)">Number of entries:
0</span></div>
<div><br>
</div>
</div>
</div>
<br>
<div> </div>
<div>running the "gluster volume heal engine" don't
solve
the problem...<br>
</div>
<div><br>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
1. What does the glustershd.log say on all 3 nodes when you run the
command? Does it complain anything about these files?<br>
2. Are these 12 files also present in the 3rd data brick?<br>
3. Can you provide the output of `gluster volume info` for the this
volume?<br>
<blockquote type="cite"
cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Some extra info:</div>
<div><br>
</div>
<div>We have recently changed the gluster from: 2 (full
repliacated) + 1 arbiter to 3 full replicated cluster </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
Just curious, how did you do this? `remove-brick` of arbiter brick
followed by an `add-brick` to increase to replica-3?<br>
<br>
Thanks,<br>
Ravi<br>
<blockquote type="cite"
cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>but i don't know this is the problem...</div>
<div><br>
</div>
<div>The "data" volume is good and healty and have no
unsynced entry.</div>
<div><br>
</div>
<div>Ovirt refuse to put the node02 and node01 in
"maintenance mode" and complains about "unsynced
elements"<br>
</div>
<div><br>
</div>
<div>How can I fix this?</div>
<div>Thank you </div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/<wbr>mai...
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>
--------------0C293E7D7C0723BAFB7F6D00--
3:50 a.m.
Hi,
Thank you for the answer and sorry for delay:
2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
1. What does the glustershd.log say on all 3 nodes when you run the
command? Does it complain anything about these files?
No, glustershd.log is clean, no extra log after command on all 3 nodes
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this
volume?
*Volume Name: engine*
*Type: Replicate*
*Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
*Status: Started*
*Snapshot Count: 0*
*Number of Bricks: 1 x 3 = 3*
*Transport-type: tcp*
*Bricks:*
*Brick1: node01:/gluster/engine/brick*
*Brick2: node02:/gluster/engine/brick*
*Brick3: node04:/gluster/engine/brick*
*Options Reconfigured:*
*nfs.disable: on*
*performance.readdir-ahead: on*
*transport.address-family: inet*
*storage.owner-uid: 36*
*performance.quick-read: off*
*performance.read-ahead: off*
*performance.io-cache: off*
*performance.stat-prefetch: off*
*performance.low-prio-threads: 32*
*network.remote-dio: off*
*cluster.eager-lock: enable*
*cluster.quorum-type: auto*
*cluster.server-quorum-type: server*
*cluster.data-self-heal-algorithm: full*
*cluster.locking-scheme: granular*
*cluster.shd-max-threads: 8*
*cluster.shd-wait-qlength: 10000*
*features.shard: on*
*user.cifs: off*
*storage.owner-gid: 36*
*features.shard-block-size: 512MB*
*network.ping-timeout: 30*
*performance.strict-o-direct: on*
*cluster.granular-entry-heal: on*
*auth.allow: **
server.allow-insecure: on
Some extra info:
>
> We have recently changed the gluster from: 2 (full repliacated) + 1
> arbiter to 3 full replicated cluster
>
Just curious, how did you do this? `remove-brick` of arbiter brick
followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick
force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick
*(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node02:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
Thanks,
Ravi
Another extra info (I don't know if this can be the problem): Five days ago
A black out has suddenly shut down the networks switch (also gluster
network) of node 03 and 04 ... But I don't know this problem is in place
after this black out
Thank you!
4:34 a.m.
This is a multi-part message in MIME format.
--------------C563F370DBD059BBFD94E893
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 07/20/2017 02:20 PM, yayo (j) wrote:
Hi,
Thank you for the answer and sorry for delay:
2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com
<mailto:ravishankar@redhat.com>>:
1. What does the glustershd.log say on all 3 nodes when you run
the command? Does it complain anything about these files?
No, glustershd.log is clean, no extra log after command on all 3 nodes
Could you check if the self-heal daemon on all nodes is connected to the
3 bricks? You will need to check the glustershd.log for that.
If it is not connected, try restarting the shd using `gluster volume
start engine force`, then launch the heal command like you did earlier
and see if heals happen.
If it doesn't, please provide the getfattr outputs of the 12 files from
all 3 nodes using `getfattr -d -m . -e hex
//gluster/engine/brick//path-to-file` ?
Thanks,
Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the
this volume?
/Volume Name: engine/
/Type: Replicate/
/Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: node01:/gluster/engine/brick/
/Brick2: node02:/gluster/engine/brick/
/Brick3: node04:/gluster/engine/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: off/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 10000/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */
server.allow-insecure: on
> Some extra info:
>
> We have recently changed the gluster from: 2 (full
> repliacated) + 1 arbiter to 3 full replicated cluster
>
Just curious, how did you do this? `remove-brick` of arbiter
brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2
node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3
node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node02:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
Thanks,
Ravi
Another extra info (I don't know if this can be the problem): Five
days ago A black out has suddenly shut down the networks switch (also
gluster network) of node 03 and 04 ... But I don't know this problem
is in place after this black out
Thank you!
--------------C563F370DBD059BBFD94E893
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/20/2017 02:20 PM, yayo (j)
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Hi, </div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Thank you for the answer and sorry
for delay:</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">2017-07-19 16:55 GMT+02:00
Ravishankar N <span dir="ltr"><<a
href="mailto:ravishankar@redhat.com" target="_blank"
moz-do-not-send="true">ravishankar@redhat.com</a>></span>:</div>
<div class="gmail_quote"><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 1. What does the glustershd.log
say on all 3 nodes when you run the command? Does it
complain anything about these files?<br>
</div>
</blockquote>
<div><br>
</div>
<div>No, glustershd.log is clean, no extra log after command
on all 3 nodes</div>
</div>
</div>
</div>
</blockquote>
<br>
Could you check if the self-heal daemon on all nodes is connected to
the 3 bricks? You will need to check the glustershd.log for that.<br>
If it is not connected, try restarting the shd using `gluster volume
start engine force`, then launch the heal command like you did
earlier and see if heals happen.<br>
<br>
If it doesn't, please provide the getfattr outputs of the 12 files
from all 3 nodes using `getfattr -d -m . -e hex
<i>/gluster/engine/brick/</i>path-to-file`
?<br>
<br>
Thanks,<br>
Ravi<br>
<br>
<blockquote type="cite"
cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 2. Are these 12 files also present
in the 3rd data brick?<br>
</div>
</blockquote>
<div><br>
</div>
<div>I've checked right now: all files exists in all 3
nodes </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 3. Can you provide the output of
`gluster volume info` for the this volume?</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume Name: engine</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Type: Replicate</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume ID:
d19c19e3-910d-437b-8ba7-4f2a23d17515</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Status: Started</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Snapshot Count: 0</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Bricks: 1 x 3 = 3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Transport-type: tcp</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Bricks:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick1:
node01:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick2:
node02:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick3:
node04:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Options Reconfigured:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>nfs.disable: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.readdir-ahead: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>transport.address-family: inet</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-uid: 36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.quick-read: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.read-ahead: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.io-cache: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.stat-prefetch: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.low-prio-threads:
32</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.remote-dio: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.eager-lock: enable</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.quorum-type: auto</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.server-quorum-type:
server</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.data-self-heal-algorithm:
full</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.locking-scheme:
granular</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-max-threads: 8</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-wait-qlength: 10000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>user.cifs: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-gid: 36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard-block-size:
512MB</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.ping-timeout: 30</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.strict-o-direct: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.granular-entry-heal: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>auth.allow: *</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div> server.allow-insecure: on</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span
class="gmail-"><br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Some extra info:</div>
<div><br>
</div>
<div>We have recently changed the gluster
from: 2 (full repliacated) + 1 arbiter to
3 full replicated cluster </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</span> Just curious, how did you do this?
`remove-brick` of arbiter brick followed by an
`add-brick` to increase to replica-3?<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes</div>
<br>
<div>
<div class="gmail-hljs gmail-nginx"
style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace">
<div><br>
</div>
<div>
<div><span
class="gmail-hljs-attribute">#gluster</span>
volume remove-brick engine replica <span
class="gmail-hljs-number"
style="color:rgb(136,0,0)">2</span>
node03:/gluster/data/brick force <b>(OK!)</b></div>
<div><br>
</div>
<div>#gluster volume heal engine <span
class="gmail-hljs-literal"
style="color:rgb(120,169,96)">info</span>
<b>(<span
class="gmail-hljs-literal"
style="color:rgb(120,169,96)">no</span>
entries!)</b></div>
<div><br>
</div>
<div>#gluster volume add-brick engine replica <span
class="gmail-hljs-number"
style="color:rgb(136,0,0)">3</span>
node04:/gluster/engine/brick <b>(OK!)</b></div>
<div><br>
</div>
<div><b>After some minutes</b></div>
<div><br>
</div>
<div>[root<span class="gmail-hljs-variable"
style="color:rgb(188,96,96)">@node01</span>
~]<span
class="gmail-hljs-comment"
style="color:rgb(136,136,136)"># gluster volume
heal engine info</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node01:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
<div><br>
</div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node02:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
<div><br>
</div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node04:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
</div>
</div>
</div>
<br>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> Thanks,<br>
Ravi</div>
</blockquote>
<div><br>
</div>
<div>Another extra info (I don't know if this can be the
problem): Five days ago A black out has suddenly shut down
the networks switch (also gluster network) of node 03 and
04 ... But I don't know this problem is in place after
this black out </div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------C563F370DBD059BBFD94E893--
5:12 a.m.
2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
Could you check if the self-heal daemon on all nodes is connected to the 3
bricks? You will need to check the glustershd.log for that.
If it is not connected, try restarting the shd using `gluster volume start
engine force`, then launch the heal command like you did earlier and see if
heals happen.
I've executed the command on all 3 nodes (Know is enougth only one) , after
that the "heal" command report elements between 6 and 10 ... (sometime 6,
sometime 8, sometime 10)
Log on glustershd.log don't say anything :
*[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
sources=[0] 1 sinks=2*
*[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81*
*[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
sources=[0] 1 sinks=2*
If it doesn't, please provide the getfattr outputs of the 12
files from
all 3 nodes using `getfattr -d -m . -e hex */gluster/engine/brick/*
path-to-file` ?
*NODE01:*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000120000000000000000*
*trusted.bit-rot.version=0x090000000000000059647d5b000447e9*
*trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x0000000e0000000000000000*
*trusted.bit-rot.version=0x090000000000000059647d5b000447e9*
*trusted.gfid=0x676067891f344c1586b8c0d05b07f187*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000550000000000000000*
*trusted.bit-rot.version=0x090000000000000059647d5b000447e9*
*trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000070000000000000000*
*trusted.bit-rot.version=0x090000000000000059647d5b000447e9*
*trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000000000000000000000*
*trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9*
*trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file: gluster/engine/brick/__DIRECT_IO_TEST__*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000000000000000000000*
*trusted.gfid=0xf05b97422771484a85fc5b6974bcef81*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000010000000000000000*
*trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9*
*trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x0000000a0000000000000000*
*trusted.bit-rot.version=0x090000000000000059647d5b000447e9*
*trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0*
*NODE02:*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x0000001a0000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48*
*trusted.afr.dirty=0x000000010000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x0000000c0000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0x676067891f344c1586b8c0d05b07f187*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-1=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x0000008e0000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000090000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000010000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file: gluster/engine/brick/__DIRECT_IO_TEST__*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000000000000000000000*
*trusted.gfid=0xf05b97422771484a85fc5b6974bcef81*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000020000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.afr.engine-client-0=0x000000000000000000000000*
*trusted.afr.engine-client-2=0x000000120000000000000000*
*trusted.bit-rot.version=0x08000000000000005965ede0000c352d*
*trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0*
*NODE04:*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0x676067891f344c1586b8c0d05b07f187*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file: gluster/engine/brick/__DIRECT_IO_TEST__*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.bit-rot.version=0x0200000000000000596484e20006237b*
*trusted.gfid=0xf05b97422771484a85fc5b6974bcef81*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.afr.dirty=0x000000000000000000000000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327*
*trusted.glusterfs.shard.block-size=0x0000000020000000*
*trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000*
*getfattr: Removing leading '/' from absolute path names*
*# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64*
*security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000*
*trusted.bit-rot.version=0x050000000000000059662c390006b836*
*trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0*
hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER
JOIN, I hope to remember) ... You think I needs to relabel? how?
*[root@node01 ~]# sestatus*
*SELinux status: disabled*
*[root@node02 ~]# sestatus*
*SELinux status: disabled*
*[root@node04 ~]# sestatus*
*SELinux status: disabled*
Thank you
Thanks,
Ravi
> 2. Are these 12 files also present in the 3rd data brick?
>
I've checked right now: all files exists in all 3 nodes
> 3. Can you provide the output of `gluster volume info` for the this
> volume?
>
*Volume Name: engine*
*Type: Replicate*
*Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
*Status: Started*
*Snapshot Count: 0*
*Number of Bricks: 1 x 3 = 3*
*Transport-type: tcp*
*Bricks:*
*Brick1: node01:/gluster/engine/brick*
*Brick2: node02:/gluster/engine/brick*
*Brick3: node04:/gluster/engine/brick*
*Options Reconfigured:*
*nfs.disable: on*
*performance.readdir-ahead: on*
*transport.address-family: inet*
*storage.owner-uid: 36*
*performance.quick-read: off*
*performance.read-ahead: off*
*performance.io-cache: off*
*performance.stat-prefetch: off*
*performance.low-prio-threads: 32*
*network.remote-dio: off*
*cluster.eager-lock: enable*
*cluster.quorum-type: auto*
*cluster.server-quorum-type: server*
*cluster.data-self-heal-algorithm: full*
*cluster.locking-scheme: granular*
*cluster.shd-max-threads: 8*
*cluster.shd-wait-qlength: 10000*
*features.shard: on*
*user.cifs: off*
*storage.owner-gid: 36*
*features.shard-block-size: 512MB*
*network.ping-timeout: 30*
*performance.strict-o-direct: on*
*cluster.granular-entry-heal: on*
*auth.allow: **
server.allow-insecure: on
>
> Some extra info:
>>
>> We have recently changed the gluster from: 2 (full repliacated) + 1
>> arbiter to 3 full replicated cluster
>>
>
> Just curious, how did you do this? `remove-brick` of arbiter brick
> followed by an `add-brick` to increase to replica-3?
>
>
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick
force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick
*(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info
Brick node01:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node02:/gluster/engine/brick
Status: Connected
Number of entries: 0
Brick node04:/gluster/engine/brick
Status: Connected
Number of entries: 0
> Thanks,
> Ravi
>
Another extra info (I don't know if this can be the problem): Five days
ago A black out has suddenly shut down the networks switch (also gluster
network) of node 03 and 04 ... But I don't know this problem is in place
after this black out
Thank you!
--
Linux User: 369739 http://counter.li.org
7:48 a.m.
This is a multi-part message in MIME format.
--------------5903BBB64769515A20658214
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 07/20/2017 03:42 PM, yayo (j) wrote:
2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com
<mailto:ravishankar@redhat.com>>:
Could you check if the self-heal daemon on all nodes is connected
to the 3 bricks? You will need to check the glustershd.log for that.
If it is not connected, try restarting the shd using `gluster
volume start engine force`, then launch the heal command like you
did earlier and see if heals happen.
I've executed the command on all 3 nodes (Know is enougth only one) ,
after that the "heal" command report elements between 6 and 10 ...
(sometime 6, sometime 8, sometime 10)
Log on glustershd.log don't say anything :
But it does say something. All these gfids of completed heals in the
log below are the for the ones that you have given the getfattr output
of. So what is likely happening is there is an intermittent connection
problem between your mount and the brick process, leading to pending
heals again after the heal gets completed, which is why the numbers are
varying each time. You would need to check why that is the case.
Hope this helps,
Ravi
/[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/
/[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81/
/[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
If it doesn't, please provide the getfattr outputs of the 12 files
from all 3 nodes using `getfattr -d -m . -e hex
//gluster/engine/brick//path-to-file` ?
*/NODE01:/*
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000120000000000000000/
/trusted.bit-rot.version=0x090000000000000059647d5b000447e9/
/trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x0000000e0000000000000000/
/trusted.bit-rot.version=0x090000000000000059647d5b000447e9/
/trusted.gfid=0x676067891f344c1586b8c0d05b07f187/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000550000000000000000/
/trusted.bit-rot.version=0x090000000000000059647d5b000447e9/
/trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000070000000000000000/
/trusted.bit-rot.version=0x090000000000000059647d5b000447e9/
/trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000000000000000000000/
/trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9/
/trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file: gluster/engine/brick/__DIRECT_IO_TEST__/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000000000000000000000/
/trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000010000000000000000/
/trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9/
/trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x0000000a0000000000000000/
/trusted.bit-rot.version=0x090000000000000059647d5b000447e9/
/trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/
/
/
*/
/*
*/NODE02:/*
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x0000001a0000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/
/trusted.afr.dirty=0x000000010000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x0000000c0000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0x676067891f344c1586b8c0d05b07f187/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-1=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x0000008e0000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000090000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000010000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file: gluster/engine/brick/__DIRECT_IO_TEST__/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000000000000000000000/
/trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000020000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.afr.engine-client-0=0x000000000000000000000000/
/trusted.afr.engine-client-2=0x000000120000000000000000/
/trusted.bit-rot.version=0x08000000000000005965ede0000c352d/
/trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/
/
/
/
/
/
/
/
/
/*NODE04*:/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0x676067891f344c1586b8c0d05b07f187/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file: gluster/engine/brick/__DIRECT_IO_TEST__/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.bit-rot.version=0x0200000000000000596484e20006237b/
/trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.afr.dirty=0x000000000000000000000000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/
/trusted.glusterfs.shard.block-size=0x0000000020000000/
/trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/
/
/
/getfattr: Removing leading '/' from absolute path names/
/# file:
gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/
/security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/
/trusted.bit-rot.version=0x050000000000000059662c390006b836/
/trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/
hum.... Is selinux the problem? but on node04 was disabled (AFTER
GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how?
/[root@node01 ~]# sestatus/
/SELinux status: disabled/
/
/
/[root@node02 ~]# sestatus/
/SELinux status: disabled/
/
/
/[root@node04 ~]# sestatus/
/SELinux status: disabled/
Thank you
Thanks,
Ravi
> 2. Are these 12 files also present in the 3rd data brick?
>
>
> I've checked right now: all files exists in all 3 nodes
>
> 3. Can you provide the output of `gluster volume info` for
> the this volume?
>
>
>
> /Volume Name: engine/
> /Type: Replicate/
> /Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/
> /Status: Started/
> /Snapshot Count: 0/
> /Number of Bricks: 1 x 3 = 3/
> /Transport-type: tcp/
> /Bricks:/
> /Brick1: node01:/gluster/engine/brick/
> /Brick2: node02:/gluster/engine/brick/
> /Brick3: node04:/gluster/engine/brick/
> /Options Reconfigured:/
> /nfs.disable: on/
> /performance.readdir-ahead: on/
> /transport.address-family: inet/
> /storage.owner-uid: 36/
> /performance.quick-read: off/
> /performance.read-ahead: off/
> /performance.io-cache: off/
> /performance.stat-prefetch: off/
> /performance.low-prio-threads: 32/
> /network.remote-dio: off/
> /cluster.eager-lock: enable/
> /cluster.quorum-type: auto/
> /cluster.server-quorum-type: server/
> /cluster.data-self-heal-algorithm: full/
> /cluster.locking-scheme: granular/
> /cluster.shd-max-threads: 8/
> /cluster.shd-wait-qlength: 10000/
> /features.shard: on/
> /user.cifs: off/
> /storage.owner-gid: 36/
> /features.shard-block-size: 512MB/
> /network.ping-timeout: 30/
> /performance.strict-o-direct: on/
> /cluster.granular-entry-heal: on/
> /auth.allow: */
>
> server.allow-insecure: on
>
>
>
>
>> Some extra info:
>>
>> We have recently changed the gluster from: 2 (full
>> repliacated) + 1 arbiter to 3 full replicated cluster
>>
>
> Just curious, how did you do this? `remove-brick` of arbiter
> brick followed by an `add-brick` to increase to replica-3?
>
>
> Yes
>
>
> #gluster volume remove-brick engine replica 2
> node03:/gluster/data/brick force *(OK!)*
>
> #gluster volume heal engine info *(no entries!)*
>
> #gluster volume add-brick engine replica 3
> node04:/gluster/engine/brick *(OK!)*
>
> *After some minutes*
>
> [root@node01 ~]# gluster volume heal engine info
> Brick node01:/gluster/engine/brick
> Status: Connected
> Number of entries: 0
>
> Brick node02:/gluster/engine/brick
> Status: Connected
> Number of entries: 0
>
> Brick node04:/gluster/engine/brick
> Status: Connected
> Number of entries: 0
>
> Thanks,
> Ravi
>
>
> Another extra info (I don't know if this can be the problem):
> Five days ago A black out has suddenly shut down the networks
> switch (also gluster network) of node 03 and 04 ... But I don't
> know this problem is in place after this black out
>
> Thank you!
>
--
Linux User: 369739 http://counter.li.org
--------------5903BBB64769515A20658214
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 07/20/2017 03:42 PM, yayo (j)
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGK=3kycgTSLxRLZSPNZi+ERh_=OhDFH29hXHzWEPnsV3vmhAQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">2017-07-20 11:34 GMT+02:00
Ravishankar N <span dir="ltr"><<a
href="mailto:ravishankar@redhat.com" target="_blank"
moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span
class="gmail-m_7883186325552308780gmail-"> <br>
<div
class="gmail-m_7883186325552308780gmail-m_9132661499198415129moz-cite-prefix">Could
you check if the self-heal daemon on all nodes is
connected to the 3 bricks? You will need to check
the glustershd.log for that.<br>
</div>
</span> If it is not connected, try restarting the shd
using `gluster volume start engine force`, then launch
the heal command like you did earlier and see if heals
happen.<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>I've executed the command on all 3 nodes (Know is
enougth only one) , after that the "heal" command report
elements between 6 and 10 ... (sometime 6, sometime 8,
sometime 10)</div>
<div><br>
</div>
<div><br>
</div>
<div>Log on glustershd.log don't say anything :</div>
</div>
</div>
</div>
</blockquote>
<br>
But it does say something. All these gfids of completed heals in
the log below are the for the ones that you have given the getfattr
output of. So what is likely happening is there is an intermittent
connection problem between your mount and the brick process, leading
to pending heals again after the heal gets completed, which is why
the numbers are varying each time. You would need to check why that
is the case.<br>
Hope this helps,<br>
Ravi<br>
<br>
<blockquote type="cite"
cite="mid:CAGK=3kycgTSLxRLZSPNZi+ERh_=OhDFH29hXHzWEPnsV3vmhAQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:<wbr>afr_log_selfheal]
0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-<wbr>7b6f5bd74327.
sources=[0] 1 sinks=2</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal
on
f05b9742-2771-484a-85fc-<wbr>5b6974bcef81</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:<wbr>afr_log_selfheal]
0-engine-replicate-0: Completed metadata selfheal on
f05b9742-2771-484a-85fc-<wbr>5b6974bcef81.
sources=[0] 1 sinks=2</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> If it doesn't, please provide
the
getfattr outputs of the 12 files from all 3 nodes using
`getfattr -d -m . -e hex
<i>/gluster/engine/brick/</i>path-to-<wbr>file`
?<br>
<br>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><b><i>NODE01:</i></b></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000120000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x0000000e0000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000550000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000070000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x0f0000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000010000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x0f0000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x0000000a0000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><b><i><br>
</i></b></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><b><i>NODE02:</i></b></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x0000001a0000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000010000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x0000000c0000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x0000008e0000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000090000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000010000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000020000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.engine-client-2=<wbr>0x000000120000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><b>NODE04</b>:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x0200000000000000596484e20006<wbr>237b</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>getfattr: Removing leading '/' from absolute
path names</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i># file:
gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>hum.... Is selinux the problem? but on node04 was
disabled (AFTER GLUSTER JOIN, I hope to remember) ... You
think I needs to relabel? how?</div>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[root@node01 ~]# sestatus</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>SELinux status:
disabled</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[root@node02 ~]# sestatus</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>SELinux status:
disabled</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div><i><br>
</i></div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[root@node04 ~]# sestatus</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>SELinux status:
disabled</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div><br>
</div>
<div>Thank you </div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> Thanks,<br>
Ravi
<div>
<div
class="gmail-m_7883186325552308780gmail-h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 2. Are these 12
files also present in the 3rd data
brick?<br>
</div>
</blockquote>
<div><br>
</div>
<div>I've checked right now: all files
exists in all 3 nodes </div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 3. Can you provide
the output of `gluster volume info` for
the this volume?</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume Name:
engine</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Type: Replicate</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume ID:
d19c19e3-910d-437b-8ba7-4f2a23<wbr>d17515</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Status: Started</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Snapshot Count:
0</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Bricks: 1 x 3 =
3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Transport-type:
tcp</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Bricks:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick1:
node01:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick2:
node02:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick3:
node04:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Options
Reconfigured:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>nfs.disable: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.readdir-ahead:
on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>transport.address-family:
inet</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-uid:
36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.quick-read:
off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.read-ahead:
off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.io-cache:
off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.stat-prefetch:
off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.low-prio-threads:
32</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.remote-dio:
off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.eager-lock:
enable</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.quorum-type:
auto</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.server-quorum-type:
server</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.data-self-heal-algorit<wbr>hm:
full</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.locking-scheme:
granular</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-max-threads:
8</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-wait-qlength:
10000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard:
on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>user.cifs: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-gid:
36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard-block-size:
512MB</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.ping-timeout:
30</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.strict-o-direct:
on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.granular-entry-heal:
on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>auth.allow: *</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div> server.allow-insecure: on</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-"><br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Some extra info:</div>
<div><br>
</div>
<div>We have recently
changed the gluster from:
2 (full repliacated) + 1
arbiter to 3 full
replicated cluster </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</span> Just curious, how did you do
this? `remove-brick` of arbiter brick
followed by an `add-brick` to increase
to replica-3?<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes</div>
<br>
<div>
<div
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs
gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-nginx"
style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace">
<div><br>
</div>
<div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-attribute">#gluster</span>
volume remove-brick engine replica <span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-number"
style="color:rgb(136,0,0)">2</span>
node03:/gluster/data/brick force
<b>(OK!)</b></div>
<div><br>
</div>
<div>#gluster volume heal engine <span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-literal"
style="color:rgb(120,169,96)">info</span>
<b>(<span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-literal"
style="color:rgb(120,169,96)">no</span>
entries!)</b></div>
<div><br>
</div>
<div>#gluster volume add-brick engine
replica <span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-number"
style="color:rgb(136,0,0)">3</span>
node04:/gluster/engine/brick
<b>(OK!)</b></div>
<div><br>
</div>
<div><b>After some
minutes</b></div>
<div><br>
</div>
<div>[root<span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-variable"
style="color:rgb(188,96,96)">@node01</span>
~]<span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">#
gluster volume heal engine
info</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node01:/gluster/engine/brick</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Number
of entries: 0</span></div>
<div><br>
</div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node02:/gluster/engine/brick</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Number
of entries: 0</span></div>
<div><br>
</div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node04:/gluster/engine/brick</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Status:
Connected</span></div>
<div><span
class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment"
style="color:rgb(136,136,136)">Number
of entries: 0</span></div>
</div>
</div>
</div>
<br>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> Thanks,<br>
Ravi</div>
</blockquote>
<div><br>
</div>
<div>Another extra info (I don't know if
this can be the problem): Five days ago A
black out has suddenly shut down the
networks switch (also gluster network) of
node 03 and 04 ... But I don't know this
problem is in place after this black out </div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail-m_7883186325552308780gmail_signature">Linux
User: 369739 <a href="http://counter.li.org"
target="_blank"
moz-do-not-send="true">http://counter.li.org</a></div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------5903BBB64769515A20658214--
4:25 a.m.
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
But it does say something. All these gfids of completed heals in the log
below are the for the ones that you have given the getfattr output of. So
what is likely happening is there is an intermittent connection problem
between your mount and the brick process, leading to pending heals again
after the heal gets completed, which is why the numbers are varying each
time. You would need to check why that is the case.
Hope this helps,
Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
sources=[0] 1 sinks=2*
*[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81*
*[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
sources=[0] 1 sinks=2*
Hi,
But we ha1e 2 gluster volume on the same network and the other one (the
"Data" gluster) don't have any problems. Why you think there is a network
problem? How to check this on a gluster infrastructure?
Thank you
6:10 a.m.
This is a multi-part message in MIME format.
--------------84EB6DEB84AAFAA57D0538D6
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 07/21/2017 02:55 PM, yayo (j) wrote:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com
<mailto:ravishankar@redhat.com>>:
But it does say something. All these gfids of completed heals in
the log below are the for the ones that you have given the
getfattr output of. So what is likely happening is there is an
intermittent connection problem between your mount and the brick
process, leading to pending heals again after the heal gets
completed, which is why the numbers are varying each time. You
would need to check why that is the case.
Hope this helps,
Ravi
>
> /[2017-07-20 09:58:46.573079] I [MSGID: 108026]
> [afr-self-heal-common.c:1254:afr_log_selfheal]
> 0-engine-replicate-0: Completed data selfheal on
> e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/
> /[2017-07-20 09:59:22.995003] I [MSGID: 108026]
> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
> 0-engine-replicate-0: performing metadata selfheal on
> f05b9742-2771-484a-85fc-5b6974bcef81/
> /[2017-07-20 09:59:22.999372] I [MSGID: 108026]
> [afr-self-heal-common.c:1254:afr_log_selfheal]
> 0-engine-replicate-0: Completed metadata selfheal on
> f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
>
Hi,
But we ha1e 2 gluster volume on the same network and the other one
(the "Data" gluster) don't have any problems. Why you think there is a
network problem?
Because pending self-heals come into the picture when I/O from the
clients (mounts) do not succeed on some bricks. They are mostly due to
(a) the client losing connection to some bricks (likely),
(b) the I/O failing on the bricks themselves (unlikely).
If most of the i/o is also going to the 3rd brick (since you say the
files are already present on all bricks and I/O is successful) , then it
is likely to be (a).
How to check this on a gluster infrastructure?
In the fuse mount logs for the engine volume, check if there are any
messages for brick disconnects. Something along the lines of
"disconnected from volname-client-x".
Just guessing here, but maybe even the 'data' volume did experience
disconnects and self-heals later but you did not observe it when you ran
heal info. See the glustershd log or mount log for for self-heal
completion messages on /0-data-replicate-0 /also.
Regards,
Ravi
Thank you
--------------84EB6DEB84AAFAA57D0538D6
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/21/2017 02:55 PM, yayo (j)
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">2017-07-20 14:48 GMT+02:00
Ravishankar N <span dir="ltr"><<a
href="mailto:ravishankar@redhat.com" target="_blank"
moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span
class="">
<p><br>
</p>
</span> But it does say something. All these gfids of
completed heals in the log below are the for the ones
that you have given the getfattr output of. So what is
likely happening is there is an intermittent connection
problem between your mount and the brick process,
leading to pending heals again after the heal gets
completed, which is why the numbers are varying each
time. You would need to check why that is the case.<br>
Hope this helps,<br>
Ravi
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20 09:58:46.573079] I
[MSGID: 108026]
[afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal]
0-engine-replicate-0: Completed data
selfheal on
e6dfd556-340b-4b76-b47b-7b6f5b<wbr>d74327.
sources=[0] 1 sinks=2</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20 09:59:22.995003] I
[MSGID: 108026]
[afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do]
0-engine-replicate-0: performing
metadata selfheal on
f05b9742-2771-484a-85fc-5b6974<wbr>bcef81</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20 09:59:22.999372] I
[MSGID: 108026]
[afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal]
0-engine-replicate-0: Completed
metadata selfheal on
f05b9742-2771-484a-85fc-5b6974<wbr>bcef81.
sources=[0] 1 sinks=2</i></div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
<div class="gmail_signature"
data-smartmail="gmail_signature">Hi,</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
<div class="gmail_signature"
data-smartmail="gmail_signature">But
we ha1e 2 gluster volume on the same network and the other
one (the "Data" gluster) don't have any problems. Why you
think there is a network problem?</div>
</div>
</div>
</blockquote>
<br>
Because pending self-heals come into the picture when I/O from the
clients (mounts) do not succeed on some bricks. They are mostly due
to <br>
(a) the client losing connection to some bricks (likely),<br>
(b) the I/O failing on the bricks themselves (unlikely).<br>
<br>
If most of the i/o is also going to the 3rd brick (since you say the
files are already present on all bricks and I/O is successful) ,
then it is likely to be (a).<br>
<br>
<blockquote type="cite"
cite="mid:CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_signature"
data-smartmail="gmail_signature">
How to check this on a gluster infrastructure?</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
</div>
</div>
</blockquote>
In the fuse mount logs for the engine volume, check if there are any
messages for brick disconnects. Something along the lines of
"disconnected from volname-client-x".<br>
Just guessing here, but maybe even the 'data' volume did experience
disconnects and self-heals later but you did not observe it when you
ran heal info. See the glustershd log or mount log for for self-heal
completion messages on <i> 0-data-replicate-0 </i>also.<br>
<br>
Regards,<br>
Ravi<br>
<blockquote type="cite"
cite="mid:CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_signature"
data-smartmail="gmail_signature">Thank
you</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
<div class="gmail_signature"
data-smartmail="gmail_signature"><br>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------84EB6DEB84AAFAA57D0538D6--
12:13 p.m.
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
But it does say something. All these gfids of completed heals in the log
below are the for the ones that you have given the getfattr output of. So
what is likely happening is there is an intermittent connection problem
between your mount and the brick process, leading to pending heals again
after the heal gets completed, which is why the numbers are varying each
time. You would need to check why that is the case.
Hope this helps,
Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
sources=[0] 1 sinks=2*
*[2017-07-20 09:59:22.995003] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81*
*[2017-07-20 09:59:22.999372] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
sources=[0] 1 sinks=2*
Hi,
following your suggestion, I've checked the "peer" status and I found that
there is too many name for the hosts, I don't know if this can be the
problem or part of it:
*gluster peer status on NODE01:*
*Number of Peers: 2*
*Hostname: dnode02.localdomain.local*
*Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
*State: Peer in Cluster (Connected)*
*Other names:*
*192.168.10.52*
*dnode02.localdomain.local*
*10.10.20.90*
*10.10.10.20*
*gluster peer status on NODE02:*
*Number of Peers: 2*
*Hostname: dnode01.localdomain.local*
*Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
*State: Peer in Cluster (Connected)*
*Other names:*
*gdnode01*
*10.10.10.10*
*Hostname: gdnode04*
*Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828*
*State: Peer in Cluster (Connected)*
*Other names:*
*192.168.10.54*
*10.10.10.40*
*gluster peer status on NODE04:*
*Number of Peers: 2*
*Hostname: dnode02.neridom.dom*
*Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
*State: Peer in Cluster (Connected)*
*Other names:*
*10.10.20.90*
*gdnode02*
*192.168.10.52*
*10.10.10.20*
*Hostname: dnode01.localdomain.local*
*Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
*State: Peer in Cluster (Connected)*
*Other names:*
*gdnode01*
*10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but, only
the 10.10.10.0 network is the decidated network for gluster (rosolved
using gdnode* host names) ... You think that remove other entries can fix
the problem? So, sorry, but, how can I remove other entries?
And, what about the selinux?
Thank you
1:11 p.m.
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found
that ovirt report the "engine" volume as an "arbiter" configuration
and the
"data" volume as full replicated volume. Check these screenshots:
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=s...
But the "gluster volume info" command report that all 2 volume are full
replicated:
*Volume Name: data*
*Type: Replicate*
*Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d*
*Status: Started*
*Snapshot Count: 0*
*Number of Bricks: 1 x 3 = 3*
*Transport-type: tcp*
*Bricks:*
*Brick1: gdnode01:/gluster/data/brick*
*Brick2: gdnode02:/gluster/data/brick*
*Brick3: gdnode04:/gluster/data/brick*
*Options Reconfigured:*
*nfs.disable: on*
*performance.readdir-ahead: on*
*transport.address-family: inet*
*storage.owner-uid: 36*
*performance.quick-read: off*
*performance.read-ahead: off*
*performance.io-cache: off*
*performance.stat-prefetch: off*
*performance.low-prio-threads: 32*
*network.remote-dio: enable*
*cluster.eager-lock: enable*
*cluster.quorum-type: auto*
*cluster.server-quorum-type: server*
*cluster.data-self-heal-algorithm: full*
*cluster.locking-scheme: granular*
*cluster.shd-max-threads: 8*
*cluster.shd-wait-qlength: 10000*
*features.shard: on*
*user.cifs: off*
*storage.owner-gid: 36*
*features.shard-block-size: 512MB*
*network.ping-timeout: 30*
*performance.strict-o-direct: on*
*cluster.granular-entry-heal: on*
*auth.allow: **
*server.allow-insecure: on*
*Volume Name: engine*
*Type: Replicate*
*Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
*Status: Started*
*Snapshot Count: 0*
*Number of Bricks: 1 x 3 = 3*
*Transport-type: tcp*
*Bricks:*
*Brick1: gdnode01:/gluster/engine/brick*
*Brick2: gdnode02:/gluster/engine/brick*
*Brick3: gdnode04:/gluster/engine/brick*
*Options Reconfigured:*
*nfs.disable: on*
*performance.readdir-ahead: on*
*transport.address-family: inet*
*storage.owner-uid: 36*
*performance.quick-read: off*
*performance.read-ahead: off*
*performance.io-cache: off*
*performance.stat-prefetch: off*
*performance.low-prio-threads: 32*
*network.remote-dio: off*
*cluster.eager-lock: enable*
*cluster.quorum-type: auto*
*cluster.server-quorum-type: server*
*cluster.data-self-heal-algorithm: full*
*cluster.locking-scheme: granular*
*cluster.shd-max-threads: 8*
*cluster.shd-wait-qlength: 10000*
*features.shard: on*
*user.cifs: off*
*storage.owner-gid: 36*
*features.shard-block-size: 512MB*
*network.ping-timeout: 30*
*performance.strict-o-direct: on*
*cluster.granular-entry-heal: on*
*auth.allow: **
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz(a)gmail.com>:
2017-07-20 14:48 GMT+02:00 Ravishankar N
<ravishankar(a)redhat.com>:
>
> But it does say something. All these gfids of completed heals in the log
> below are the for the ones that you have given the getfattr output of. So
> what is likely happening is there is an intermittent connection problem
> between your mount and the brick process, leading to pending heals again
> after the heal gets completed, which is why the numbers are varying each
> time. You would need to check why that is the case.
> Hope this helps,
> Ravi
>
>
>
> *[2017-07-20 09:58:46.573079] I [MSGID: 108026]
> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
> Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
> sources=[0] 1 sinks=2*
> *[2017-07-20 09:59:22.995003] I [MSGID: 108026]
> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
> 0-engine-replicate-0: performing metadata selfheal on
> f05b9742-2771-484a-85fc-5b6974bcef81*
> *[2017-07-20 09:59:22.999372] I [MSGID: 108026]
> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
> Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
> sources=[0] 1 sinks=2*
>
>
Hi,
following your suggestion, I've checked the "peer" status and I found that
there is too many name for the hosts, I don't know if this can be the
problem or part of it:
*gluster peer status on NODE01:*
*Number of Peers: 2*
*Hostname: dnode02.localdomain.local*
*Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
*State: Peer in Cluster (Connected)*
*Other names:*
*192.168.10.52*
*dnode02.localdomain.local*
*10.10.20.90*
*10.10.10.20*
*gluster peer status on NODE02:*
*Number of Peers: 2*
*Hostname: dnode01.localdomain.local*
*Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
*State: Peer in Cluster (Connected)*
*Other names:*
*gdnode01*
*10.10.10.10*
*Hostname: gdnode04*
*Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828*
*State: Peer in Cluster (Connected)*
*Other names:*
*192.168.10.54*
*10.10.10.40*
*gluster peer status on NODE04:*
*Number of Peers: 2*
*Hostname: dnode02.neridom.dom*
*Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
*State: Peer in Cluster (Connected)*
*Other names:*
*10.10.20.90*
*gdnode02*
*192.168.10.52*
*10.10.10.20*
*Hostname: dnode01.localdomain.local*
*Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
*State: Peer in Cluster (Connected)*
*Other names:*
*gdnode01*
*10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but,
only the 10.10.10.0 network is the decidated network for gluster (rosolved
using gdnode* host names) ... You think that remove other entries can fix
the problem? So, sorry, but, how can I remove other entries?
And, what about the selinux?
Thank you
--
Linux User: 369739 http://counter.li.org
1:13 a.m.
This is a multi-part message in MIME format.
--------------6CE85A48A5119C9A9913FFCB
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've
found that ovirt report the "engine" volume as an "arbiter"
configuration and the "data" volume as full replicated volume. Check
these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to
tell you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=s...
But the "gluster volume info" command report that all 2 volume are
full replicated:
/Volume Name: data/
/Type: Replicate/
/Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: gdnode01:/gluster/data/brick/
/Brick2: gdnode02:/gluster/data/brick/
/Brick3: gdnode04:/gluster/data/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: enable/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 10000/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */
/server.allow-insecure: on/
/Volume Name: engine/
/Type: Replicate/
/Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/
/Status: Started/
/Snapshot Count: 0/
/Number of Bricks: 1 x 3 = 3/
/Transport-type: tcp/
/Bricks:/
/Brick1: gdnode01:/gluster/engine/brick/
/Brick2: gdnode02:/gluster/engine/brick/
/Brick3: gdnode04:/gluster/engine/brick/
/Options Reconfigured:/
/nfs.disable: on/
/performance.readdir-ahead: on/
/transport.address-family: inet/
/storage.owner-uid: 36/
/performance.quick-read: off/
/performance.read-ahead: off/
/performance.io-cache: off/
/performance.stat-prefetch: off/
/performance.low-prio-threads: 32/
/network.remote-dio: off/
/cluster.eager-lock: enable/
/cluster.quorum-type: auto/
/cluster.server-quorum-type: server/
/cluster.data-self-heal-algorithm: full/
/cluster.locking-scheme: granular/
/cluster.shd-max-threads: 8/
/cluster.shd-wait-qlength: 10000/
/features.shard: on/
/user.cifs: off/
/storage.owner-gid: 36/
/features.shard-block-size: 512MB/
/network.ping-timeout: 30/
/performance.strict-o-direct: on/
/cluster.granular-entry-heal: on/
/auth.allow: */
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz(a)gmail.com
<mailto:jaganz@gmail.com>>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com
<mailto:ravishankar@redhat.com>>:
But it does say something. All these gfids of completed heals
in the log below are the for the ones that you have given the
getfattr output of. So what is likely happening is there is an
intermittent connection problem between your mount and the
brick process, leading to pending heals again after the heal
gets completed, which is why the numbers are varying each
time. You would need to check why that is the case.
Hope this helps,
Ravi
>
> /[2017-07-20 09:58:46.573079] I [MSGID: 108026]
> [afr-self-heal-common.c:1254:afr_log_selfheal]
> 0-engine-replicate-0: Completed data selfheal on
> e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/
> /[2017-07-20 09:59:22.995003] I [MSGID: 108026]
> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
> 0-engine-replicate-0: performing metadata selfheal on
> f05b9742-2771-484a-85fc-5b6974bcef81/
> /[2017-07-20 09:59:22.999372] I [MSGID: 108026]
> [afr-self-heal-common.c:1254:afr_log_selfheal]
> 0-engine-replicate-0: Completed metadata selfheal on
> f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
>
Hi,
following your suggestion, I've checked the "peer" status and I
found that there is too many name for the hosts, I don't know if
this can be the problem or part of it:
/*gluster peer status on NODE01:*/
/Number of Peers: 2/
/
/
/Hostname: dnode02.localdomain.local/
/Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd/
/State: Peer in Cluster (Connected)/
/Other names:/
/192.168.10.52/
/dnode02.localdomain.local/
/10.10.20.90/
/10.10.10.20/
/
/
/
/
/
/
/
/
*/gluster peer status on //NODE02:/*
/Number of Peers: 2/
/
/
/Hostname: dnode01.localdomain.local/
/Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12/
/State: Peer in Cluster (Connected)/
/Other names:/
/gdnode01/
/10.10.10.10/
/
/
/Hostname: gdnode04/
/Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828/
/State: Peer in Cluster (Connected)/
/Other names:/
/192.168.10.54/
/10.10.10.40/
/
/
/*
*/
*/gluster peer status on //NODE04:/*
/Number of Peers: 2/
/
/
/Hostname: dnode02.neridom.dom/
/Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd/
/State: Peer in Cluster (Connected)/
/Other names:/
/10.10.20.90/
/gdnode02/
/192.168.10.52/
/10.10.10.20/
/
/
/Hostname: dnode01.localdomain.local/
/Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12/
/State: Peer in Cluster (Connected)/
/Other names:/
/gdnode01/
/10.10.10.10/
/
/
/
/
All these ip are pingable and hosts resolvible across all 3 nodes
but, only the 10.10.10.0 network is the decidated network for
gluster (rosolved using gdnode* host names) ... You think that
remove other entries can fix the problem? So, sorry, but, how can
I remove other entries?
I don't think having extra entries could be a problem. Did you check the
fuse mount logs for disconnect messages that I referred to in the other
email?
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount
logs first.
-Ravi
Thank you
--
Linux User: 369739 http://counter.li.org
--------------6CE85A48A5119C9A9913FFCB
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/21/2017 11:41 PM, yayo (j)
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com">
<div dir="ltr">Hi,
<div><br>
</div>
<div>Sorry for follow up again, but, checking the ovirt
interface I've found that ovirt report the "engine" volume as
an "arbiter" configuration and the "data" volume as full
replicated volume. Check these screenshots:</div>
</div>
</blockquote>
<br>
This is probably some refresh bug in the UI, Sahina might be able to
tell you.<br>
<blockquote type="cite"
cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com">
<div dir="ltr">
<div><br>
</div>
<div><a
href="https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVm...
moz-do-not-send="true">https://drive.google.com/drive/folder...
</div>
<div><br>
</div>
<div>But the "gluster volume info" command report that all 2
volume are full replicated:</div>
<div><br>
</div>
<div><br>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div>
<div><i>Volume Name: data</i></div>
</div>
<div>
<div><i>Type: Replicate</i></div>
</div>
<div>
<div><i>Volume ID:
c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d</i></div>
</div>
<div>
<div><i>Status: Started</i></div>
</div>
<div>
<div><i>Snapshot Count: 0</i></div>
</div>
<div>
<div><i>Number of Bricks: 1 x 3 = 3</i></div>
</div>
<div>
<div><i>Transport-type: tcp</i></div>
</div>
<div>
<div><i>Bricks:</i></div>
</div>
<div>
<div><i>Brick1:
gdnode01:/gluster/data/brick</i></div>
</div>
<div>
<div><i>Brick2:
gdnode02:/gluster/data/brick</i></div>
</div>
<div>
<div><i>Brick3:
gdnode04:/gluster/data/brick</i></div>
</div>
<div>
<div><i>Options Reconfigured:</i></div>
</div>
<div>
<div><i>nfs.disable: on</i></div>
</div>
<div>
<div><i>performance.readdir-ahead: on</i></div>
</div>
<div>
<div><i>transport.address-family: inet</i></div>
</div>
<div>
<div><i>storage.owner-uid: 36</i></div>
</div>
<div>
<div><i>performance.quick-read: off</i></div>
</div>
<div>
<div><i>performance.read-ahead: off</i></div>
</div>
<div>
<div><i>performance.io-cache: off</i></div>
</div>
<div>
<div><i>performance.stat-prefetch: off</i></div>
</div>
<div>
<div><i>performance.low-prio-threads: 32</i></div>
</div>
<div>
<div><i>network.remote-dio: enable</i></div>
</div>
<div>
<div><i>cluster.eager-lock: enable</i></div>
</div>
<div>
<div><i>cluster.quorum-type: auto</i></div>
</div>
<div>
<div><i>cluster.server-quorum-type: server</i></div>
</div>
<div>
<div><i>cluster.data-self-heal-algorithm:
full</i></div>
</div>
<div>
<div><i>cluster.locking-scheme: granular</i></div>
</div>
<div>
<div><i>cluster.shd-max-threads: 8</i></div>
</div>
<div>
<div><i>cluster.shd-wait-qlength: 10000</i></div>
</div>
<div>
<div><i>features.shard: on</i></div>
</div>
<div>
<div><i>user.cifs: off</i></div>
</div>
<div>
<div><i>storage.owner-gid: 36</i></div>
</div>
<div>
<div><i>features.shard-block-size: 512MB</i></div>
</div>
<div>
<div><i>network.ping-timeout: 30</i></div>
</div>
<div>
<div><i>performance.strict-o-direct: on</i></div>
</div>
<div>
<div><i>cluster.granular-entry-heal: on</i></div>
</div>
<div>
<div><i>auth.allow: *</i></div>
</div>
<div>
<div><i>server.allow-insecure: on</i></div>
</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<blockquote
style="color:rgb(0,0,0);font-size:12.8px;margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote"><i>Volume Name:
engine</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Type:
Replicate</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Volume ID:
d19c19e3-910d-437b-8ba7-<wbr>4f2a23d17515</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Status:
Started</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Snapshot Count:
0</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Number of Bricks: 1 x 3 =
3</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Transport-type:
tcp</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>Bricks:</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Brick1:
gdnode01:/gluster/engine/brick</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Brick2:
gdnode02:/gluster/engine/brick</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Brick3:
gdnode04:/gluster/engine/brick</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>Options
Reconfigured:</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>nfs.disable:
on</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>performance.readdir-ahead:
on</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>transport.address-family:
inet</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>storage.owner-uid:
36</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>performance.quick-read:
off</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>performance.read-ahead:
off</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>performance.io-cache:
off</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>performance.stat-prefetch:
off</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>performance.low-prio-threads:
32</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>network.remote-dio:
off</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>cluster.eager-lock:
enable</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>cluster.quorum-type:
auto</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>cluster.server-quorum-type:
server</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>cluster.data-self-heal-<wbr>algorithm:
full</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>cluster.locking-scheme:
granular</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>cluster.shd-max-threads:
8</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>cluster.shd-wait-qlength:
10000</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>features.shard:
on</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>user.cifs:
off</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>storage.owner-gid:
36</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>features.shard-block-size:
512MB</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>network.ping-timeout:
30</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>performance.strict-o-direct:
on</i></div>
</div>
<div class="gmail_extra">
<div
class="gmail_quote"><i>cluster.granular-entry-heal:
on</i></div>
</div>
<div class="gmail_extra">
<div class="gmail_quote"><i>auth.allow:
*</i></div>
</div>
</blockquote>
<div class="gmail_extra"
style="color:rgb(0,0,0);font-size:12.8px">
<div class="gmail_quote"> server.allow-insecure:
on</div>
</div>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2017-07-21 19:13 GMT+02:00 yayo (j)
<span
dir="ltr"><<a href="mailto:jaganz@gmail.com"
target="_blank"
moz-do-not-send="true">jaganz@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><span
class="">2017-07-20 14:48
GMT+02:00 Ravishankar N <span dir="ltr"><<a
href="mailto:ravishankar@redhat.com"
target="_blank"
moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span
class="m_7510525959629396992gmail-">
<p><br>
</p>
</span> But it does say something. All these
gfids of completed heals in the log below are
the for the ones that you have given the
getfattr output of. So what is likely happening
is there is an intermittent connection problem
between your mount and the brick process,
leading to pending heals again after the heal
gets completed, which is why the numbers are
varying each time. You would need to check why
that is the case.<br>
Hope this helps,<br>
Ravi
<div>
<div
class="m_7510525959629396992gmail-h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20
09:58:46.573079] I [MSGID:
108026]
[afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal]
0-engine-replicate-0:
Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5b<wbr>d74327.
sources=[0] 1 sinks=2</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20
09:59:22.995003] I [MSGID:
108026]
[afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do]
0-engine-replicate-0:
performing metadata selfheal
on
f05b9742-2771-484a-85fc-5b6974<wbr>bcef81</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>[2017-07-20
09:59:22.999372] I [MSGID:
108026]
[afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal]
0-engine-replicate-0:
Completed metadata selfheal
on
f05b9742-2771-484a-85fc-5b6974<wbr>bcef81.
sources=[0] 1 sinks=2</i></div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</span>
<div>Hi,</div>
<div><br>
</div>
<div>following your suggestion, I've checked the
"peer" status and I found that there is too many
name for the hosts, I don't know if this can be the
problem or part of it:</div>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0 0 0
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><b>gluster peer status on
NODE01:</b></i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Peers: 2</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Hostname:
dnode02.localdomain.local</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Uuid:
7c0ebfa3-5676-4d3f-9bfa-<wbr>7fff6afea0dd</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>State: Peer in Cluster
(Connected)</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Other names:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>192.168.10.52</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>dnode02.localdomain.local</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.20.90</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.10.20</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><b><i>gluster peer status
on </i><i>NODE02:</i></b></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Peers: 2</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Hostname:
dnode01.localdomain.local</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Uuid:
a568bd60-b3e4-4432-a9bc-<wbr>996c52eaaa12</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>State: Peer in Cluster
(Connected)</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Other names:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>gdnode01</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.10.10</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Hostname: gdnode04</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Uuid:
ce6e0f6b-12cf-4e40-8f01-<wbr>d1609dfc5828</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>State: Peer in Cluster
(Connected)</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Other names:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>192.168.10.54</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.10.40</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><b><br>
</b></i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><b><i>gluster peer status
on </i><i>NODE04:</i></b></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Peers: 2</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Hostname:
dnode02.neridom.dom</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Uuid:
7c0ebfa3-5676-4d3f-9bfa-<wbr>7fff6afea0dd</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>State: Peer in Cluster
(Connected)</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Other names:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.20.90</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>gdnode02</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>192.168.10.52</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.10.20</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Hostname:
dnode01.localdomain.local</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Uuid:
a568bd60-b3e4-4432-a9bc-<wbr>996c52eaaa12</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>State: Peer in Cluster
(Connected)</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Other names:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>gdnode01</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>10.10.10.10</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i><br>
</i></div>
<div><i><br>
</i></div>
<div>All these ip are pingable and hosts resolvible
across all 3 nodes but, only the 10.10.10.0
network is the decidated network for gluster
(rosolved using gdnode* host names) ... You think
that remove other entries can fix the problem? So,
sorry, but, how can I remove other entries? <br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
I don't think having extra entries could be a problem. Did you check
the fuse mount logs for disconnect messages that I referred to in
the other email?<br>
<blockquote type="cite"
cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><br>
</div>
</div>
<div>And, what about the selinux? <br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
Not sure about this. See if there are disconnect messages in the
mount logs first.<br>
-Ravi<br>
<blockquote type="cite"
cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Thank you</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature"
data-smartmail="gmail_signature">Linux
User: 369739 <a href="http://counter.li.org"
target="_blank"
moz-do-not-send="true">http://counter.li.org</a></div>
</div>
</blockquote>
<br>
</body>
</html>
--------------6CE85A48A5119C9A9913FFCB--
1:12 a.m.
Hi,
Regarding the UI showing incorrect information about engine and data
volumes, can you please refresh the UI and see if the issue persists plus
any errors in the engine.log files ?
Thanks
kasturi
On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar(a)redhat.com>
wrote:
On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found
that ovirt report the "engine" volume as an "arbiter" configuration
and the
"data" volume as full replicated volume. Check these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to tell
you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?
usp=sharing
But the "gluster volume info" command report that all 2 volume are full
replicated:
*Volume Name: data*
*Type: Replicate*
*Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d*
*Status: Started*
*Snapshot Count: 0*
*Number of Bricks: 1 x 3 = 3*
*Transport-type: tcp*
*Bricks:*
*Brick1: gdnode01:/gluster/data/brick*
*Brick2: gdnode02:/gluster/data/brick*
*Brick3: gdnode04:/gluster/data/brick*
*Options Reconfigured:*
*nfs.disable: on*
*performance.readdir-ahead: on*
*transport.address-family: inet*
*storage.owner-uid: 36*
*performance.quick-read: off*
*performance.read-ahead: off*
*performance.io-cache: off*
*performance.stat-prefetch: off*
*performance.low-prio-threads: 32*
*network.remote-dio: enable*
*cluster.eager-lock: enable*
*cluster.quorum-type: auto*
*cluster.server-quorum-type: server*
*cluster.data-self-heal-algorithm: full*
*cluster.locking-scheme: granular*
*cluster.shd-max-threads: 8*
*cluster.shd-wait-qlength: 10000*
*features.shard: on*
*user.cifs: off*
*storage.owner-gid: 36*
*features.shard-block-size: 512MB*
*network.ping-timeout: 30*
*performance.strict-o-direct: on*
*cluster.granular-entry-heal: on*
*auth.allow: **
*server.allow-insecure: on*
*Volume Name: engine*
*Type: Replicate*
*Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
*Status: Started*
*Snapshot Count: 0*
*Number of Bricks: 1 x 3 = 3*
*Transport-type: tcp*
*Bricks:*
*Brick1: gdnode01:/gluster/engine/brick*
*Brick2: gdnode02:/gluster/engine/brick*
*Brick3: gdnode04:/gluster/engine/brick*
*Options Reconfigured:*
*nfs.disable: on*
*performance.readdir-ahead: on*
*transport.address-family: inet*
*storage.owner-uid: 36*
*performance.quick-read: off*
*performance.read-ahead: off*
*performance.io-cache: off*
*performance.stat-prefetch: off*
*performance.low-prio-threads: 32*
*network.remote-dio: off*
*cluster.eager-lock: enable*
*cluster.quorum-type: auto*
*cluster.server-quorum-type: server*
*cluster.data-self-heal-algorithm: full*
*cluster.locking-scheme: granular*
*cluster.shd-max-threads: 8*
*cluster.shd-wait-qlength: 10000*
*features.shard: on*
*user.cifs: off*
*storage.owner-gid: 36*
*features.shard-block-size: 512MB*
*network.ping-timeout: 30*
*performance.strict-o-direct: on*
*cluster.granular-entry-heal: on*
*auth.allow: **
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz(a)gmail.com>:
> 2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
>
>>
>> But it does say something. All these gfids of completed heals in the
>> log below are the for the ones that you have given the getfattr output of.
>> So what is likely happening is there is an intermittent connection problem
>> between your mount and the brick process, leading to pending heals again
>> after the heal gets completed, which is why the numbers are varying each
>> time. You would need to check why that is the case.
>> Hope this helps,
>> Ravi
>>
>>
>>
>> *[2017-07-20 09:58:46.573079] I [MSGID: 108026]
>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>> Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
>> sources=[0] 1 sinks=2*
>> *[2017-07-20 09:59:22.995003] I [MSGID: 108026]
>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>> 0-engine-replicate-0: performing metadata selfheal on
>> f05b9742-2771-484a-85fc-5b6974bcef81*
>> *[2017-07-20 09:59:22.999372] I [MSGID: 108026]
>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>> Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
>> sources=[0] 1 sinks=2*
>>
>>
>
> Hi,
>
> following your suggestion, I've checked the "peer" status and I found
> that there is too many name for the hosts, I don't know if this can be the
> problem or part of it:
>
> *gluster peer status on NODE01:*
> *Number of Peers: 2*
>
> *Hostname: dnode02.localdomain.local*
> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
> *State: Peer in Cluster (Connected)*
> *Other names:*
> *192.168.10.52*
> *dnode02.localdomain.local*
> *10.10.20.90*
> *10.10.10.20*
>
>
>
>
> *gluster peer status on NODE02:*
> *Number of Peers: 2*
>
> *Hostname: dnode01.localdomain.local*
> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
> *State: Peer in Cluster (Connected)*
> *Other names:*
> *gdnode01*
> *10.10.10.10*
>
> *Hostname: gdnode04*
> *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828*
> *State: Peer in Cluster (Connected)*
> *Other names:*
> *192.168.10.54*
> *10.10.10.40*
>
>
> *gluster peer status on NODE04:*
> *Number of Peers: 2*
>
> *Hostname: dnode02.neridom.dom*
> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
> *State: Peer in Cluster (Connected)*
> *Other names:*
> *10.10.20.90*
> *gdnode02*
> *192.168.10.52*
> *10.10.10.20*
>
> *Hostname: dnode01.localdomain.local*
> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
> *State: Peer in Cluster (Connected)*
> *Other names:*
> *gdnode01*
> *10.10.10.10*
>
>
>
> All these ip are pingable and hosts resolvible across all 3 nodes but,
> only the 10.10.10.0 network is the decidated network for gluster (rosolved
> using gdnode* host names) ... You think that remove other entries can fix
> the problem? So, sorry, but, how can I remove other entries?
>
I don't think having extra entries could be a problem. Did you check the
fuse mount logs for disconnect messages that I referred to in the other
email?
> And, what about the selinux?
>
Not sure about this. See if there are disconnect messages in the mount
logs first.
-Ravi
> Thank you
>
>
>
--
Linux User: 369739 http://counter.li.org
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9 a.m.
Hi,
UI refreshed but problem still remain ...
No specific error, I've only these errors but I've read that there is no
problem if I have this kind of errors:
2017-07-24 15:53:59,823+02 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler2) [b7590c4] START,
GlusterServersListVDSCommand(HostName
= node01.localdomain.local, VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 29a62417
2017-07-24 15:54:01,066+02 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterServersListVDSCommand,
return: [10.10.20.80/24:CONNECTED, node02.localdomain.local:CONNECTED,
gdnode04:CONNECTED], log id: 29a62417
2017-07-24 15:54:01,076+02 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler2) [b7590c4] START,
GlusterVolumesListVDSCommand(HostName
= node01.localdomain.local, GlusterVolumesListVDSParameters:{runAsync='true',
hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 7fce25d3
2017-07-24 15:54:02,209+02 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler2) [b7590c4] Could not associate brick
'gdnode01:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-
4f2a23d17515' with correct network as no gluster network found in cluster
'00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,212+02 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler2) [b7590c4] Could not associate brick
'gdnode02:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-
4f2a23d17515' with correct network as no gluster network found in cluster
'00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,215+02 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler2) [b7590c4] Could not associate brick
'gdnode04:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-
4f2a23d17515' with correct network as no gluster network found in cluster
'00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,218+02 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler2) [b7590c4] Could not associate brick
'gdnode01:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-
c8275d4a7c2d' with correct network as no gluster network found in cluster
'00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,221+02 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler2) [b7590c4] Could not associate brick
'gdnode02:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-
c8275d4a7c2d' with correct network as no gluster network found in cluster
'00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,224+02 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler2) [b7590c4] Could not associate brick
'gdnode04:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-
c8275d4a7c2d' with correct network as no gluster network found in cluster
'00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,224+02 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterVolumesListVDSCommand,
return: {d19c19e3-910d-437b-8ba7-4f2a23d17515=org.ovirt.engine.core.
common.businessentities.gluster.GlusterVolumeEntity@fdc91062, c7a5dfc9-3e72
-4ea1-843e-c8275d4a7c2d=org.ovirt.engine.core.common.businessentities.
gluster.GlusterVolumeEntity@999a6f23}, log id: 7fce25d3
Thank you
2017-07-24 8:12 GMT+02:00 Kasturi Narra <knarra(a)redhat.com>:
Hi,
Regarding the UI showing incorrect information about engine and data
volumes, can you please refresh the UI and see if the issue persists plus
any errors in the engine.log files ?
Thanks
kasturi
On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar(a)redhat.com>
wrote:
>
> On 07/21/2017 11:41 PM, yayo (j) wrote:
>
> Hi,
>
> Sorry for follow up again, but, checking the ovirt interface I've found
> that ovirt report the "engine" volume as an "arbiter"
configuration and the
> "data" volume as full replicated volume. Check these screenshots:
>
>
> This is probably some refresh bug in the UI, Sahina might be able to tell
> you.
>
>
> https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFf
> VmR5aDQ?usp=sharing
>
> But the "gluster volume info" command report that all 2 volume are full
> replicated:
>
>
> *Volume Name: data*
> *Type: Replicate*
> *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d*
> *Status: Started*
> *Snapshot Count: 0*
> *Number of Bricks: 1 x 3 = 3*
> *Transport-type: tcp*
> *Bricks:*
> *Brick1: gdnode01:/gluster/data/brick*
> *Brick2: gdnode02:/gluster/data/brick*
> *Brick3: gdnode04:/gluster/data/brick*
> *Options Reconfigured:*
> *nfs.disable: on*
> *performance.readdir-ahead: on*
> *transport.address-family: inet*
> *storage.owner-uid: 36*
> *performance.quick-read: off*
> *performance.read-ahead: off*
> *performance.io-cache: off*
> *performance.stat-prefetch: off*
> *performance.low-prio-threads: 32*
> *network.remote-dio: enable*
> *cluster.eager-lock: enable*
> *cluster.quorum-type: auto*
> *cluster.server-quorum-type: server*
> *cluster.data-self-heal-algorithm: full*
> *cluster.locking-scheme: granular*
> *cluster.shd-max-threads: 8*
> *cluster.shd-wait-qlength: 10000*
> *features.shard: on*
> *user.cifs: off*
> *storage.owner-gid: 36*
> *features.shard-block-size: 512MB*
> *network.ping-timeout: 30*
> *performance.strict-o-direct: on*
> *cluster.granular-entry-heal: on*
> *auth.allow: **
> *server.allow-insecure: on*
>
>
>
>
>
> *Volume Name: engine*
> *Type: Replicate*
> *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
> *Status: Started*
> *Snapshot Count: 0*
> *Number of Bricks: 1 x 3 = 3*
> *Transport-type: tcp*
> *Bricks:*
> *Brick1: gdnode01:/gluster/engine/brick*
> *Brick2: gdnode02:/gluster/engine/brick*
> *Brick3: gdnode04:/gluster/engine/brick*
> *Options Reconfigured:*
> *nfs.disable: on*
> *performance.readdir-ahead: on*
> *transport.address-family: inet*
> *storage.owner-uid: 36*
> *performance.quick-read: off*
> *performance.read-ahead: off*
> *performance.io-cache: off*
> *performance.stat-prefetch: off*
> *performance.low-prio-threads: 32*
> *network.remote-dio: off*
> *cluster.eager-lock: enable*
> *cluster.quorum-type: auto*
> *cluster.server-quorum-type: server*
> *cluster.data-self-heal-algorithm: full*
> *cluster.locking-scheme: granular*
> *cluster.shd-max-threads: 8*
> *cluster.shd-wait-qlength: 10000*
> *features.shard: on*
> *user.cifs: off*
> *storage.owner-gid: 36*
> *features.shard-block-size: 512MB*
> *network.ping-timeout: 30*
> *performance.strict-o-direct: on*
> *cluster.granular-entry-heal: on*
> *auth.allow: **
>
> server.allow-insecure: on
>
>
> 2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz(a)gmail.com>:
>
>> 2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
>>
>>>
>>> But it does say something. All these gfids of completed heals in the
>>> log below are the for the ones that you have given the getfattr output of.
>>> So what is likely happening is there is an intermittent connection problem
>>> between your mount and the brick process, leading to pending heals again
>>> after the heal gets completed, which is why the numbers are varying each
>>> time. You would need to check why that is the case.
>>> Hope this helps,
>>> Ravi
>>>
>>>
>>>
>>> *[2017-07-20 09:58:46.573079] I [MSGID: 108026]
>>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>>> Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
>>> sources=[0] 1 sinks=2*
>>> *[2017-07-20 09:59:22.995003] I [MSGID: 108026]
>>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>>> 0-engine-replicate-0: performing metadata selfheal on
>>> f05b9742-2771-484a-85fc-5b6974bcef81*
>>> *[2017-07-20 09:59:22.999372] I [MSGID: 108026]
>>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>>> Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
>>> sources=[0] 1 sinks=2*
>>>
>>>
>>
>> Hi,
>>
>> following your suggestion, I've checked the "peer" status and I
found
>> that there is too many name for the hosts, I don't know if this can be the
>> problem or part of it:
>>
>> *gluster peer status on NODE01:*
>> *Number of Peers: 2*
>>
>> *Hostname: dnode02.localdomain.local*
>> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
>> *State: Peer in Cluster (Connected)*
>> *Other names:*
>> *192.168.10.52*
>> *dnode02.localdomain.local*
>> *10.10.20.90*
>> *10.10.10.20*
>>
>>
>>
>>
>> *gluster peer status on NODE02:*
>> *Number of Peers: 2*
>>
>> *Hostname: dnode01.localdomain.local*
>> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
>> *State: Peer in Cluster (Connected)*
>> *Other names:*
>> *gdnode01*
>> *10.10.10.10*
>>
>> *Hostname: gdnode04*
>> *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828*
>> *State: Peer in Cluster (Connected)*
>> *Other names:*
>> *192.168.10.54*
>> *10.10.10.40*
>>
>>
>> *gluster peer status on NODE04:*
>> *Number of Peers: 2*
>>
>> *Hostname: dnode02.neridom.dom*
>> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
>> *State: Peer in Cluster (Connected)*
>> *Other names:*
>> *10.10.20.90*
>> *gdnode02*
>> *192.168.10.52*
>> *10.10.10.20*
>>
>> *Hostname: dnode01.localdomain.local*
>> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
>> *State: Peer in Cluster (Connected)*
>> *Other names:*
>> *gdnode01*
>> *10.10.10.10*
>>
>>
>>
>> All these ip are pingable and hosts resolvible across all 3 nodes but,
>> only the 10.10.10.0 network is the decidated network for gluster (rosolved
>> using gdnode* host names) ... You think that remove other entries can fix
>> the problem? So, sorry, but, how can I remove other entries?
>>
> I don't think having extra entries could be a problem. Did you check the
> fuse mount logs for disconnect messages that I referred to in the other
> email?
>
>
>> And, what about the selinux?
>>
> Not sure about this. See if there are disconnect messages in the mount
> logs first.
> -Ravi
>
>
>> Thank you
>>
>>
>>
>
>
> --
> Linux User: 369739 http://counter.li.org
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
--
Linux User: 369739 http://counter.li.org
9:16 a.m.
All these ip are pingable and hosts resolvible across all 3 nodes but,
> only the 10.10.10.0 network is the decidated network for gluster (rosolved
> using gdnode* host names) ... You think that remove other entries can fix
> the problem? So, sorry, but, how can I remove other entries?
>
I don't think having extra entries could be a problem. Did you check the
fuse mount logs for disconnect messages that I referred to in the other
email?
* tail -f
/var/log/glusterfs/rhev-data-center-mnt-glusterSD-dvirtgluster\:engine.log*
*NODE01:*
[2017-07-24 07:34:00.799347] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0
-glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport
endpoint is not connected)
[2017-07-24 07:44:46.687334] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0
-glusterfsd-mgmt: Exhausted all volfile servers
[2017-07-24 09:04:25.951350] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0
-glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport
endpoint is not connected)
[2017-07-24 09:15:11.839357] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0
-glusterfsd-mgmt: Exhausted all volfile servers
[2017-07-24 10:34:51.231353] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0
-glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport
endpoint is not connected)
[2017-07-24 10:45:36.991321] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0
-glusterfsd-mgmt: Exhausted all volfile servers
[2017-07-24 12:05:16.383323] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0
-glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport
endpoint is not connected)
[2017-07-24 12:16:02.271320] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0
-glusterfsd-mgmt: Exhausted all volfile servers
[2017-07-24 13:35:41.535308] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0
-glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport
endpoint is not connected)
[2017-07-24 13:46:27.423304] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0
-glusterfsd-mgmt: Exhausted all volfile servers
Why again gdnode03? Was removed from gluster! was the arbiter node...
*NODE02:*
[2017-07-24 14:08:18.709209] I [MSGID: 108026] [
afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on db56ac00-fd5b-4326-a879-326ff56181de. sources=0 [
1] sinks=2
[2017-07-24 14:08:38.746688] I [MSGID: 108026] [
afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81
[2017-07-24 14:08:38.749379] I [MSGID: 108026] [
afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
sources=0 [1] sinks=2
[2017-07-24 14:08:46.068001] I [MSGID: 108026] [
afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on db56ac00-fd5b-4326-a879-326ff56181de. sources=0 [
1] sinks=2
The message "I [MSGID: 108026] [
afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-engine-replicate-0: performing metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81" repeated 3 times between [2017-07-24
14:08:38.746688] and [2017-07-24 14:10:09.088625]
The message "I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal]
0-engine-replicate-0: Completed metadata selfheal on
f05b9742-2771-484a-85fc-5b6974bcef81. sources=0 [1] sinks=2 " repeated 3
times between [2017-07-24 14:08:38.749379] and [2017-07-24 14:10:09.091377]
[2017-07-24 14:10:19.384379] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on db56ac00-fd5b-4326-a879-326ff56181de. sources=0
[1] sinks=2
[2017-07-24 14:10:39.433155] I [MSGID: 108026] [afr-self-heal-metadata.c:51:
__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata
selfheal on f05b9742-2771-484a-85fc-5b6974bcef81
[2017-07-24 14:10:39.435847] I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
sources=0 [1] sinks=2
*NODE04:*
[2017-07-24 14:08:56.789598] I [MSGID: 108026] [afr-self-heal-common.c:1254
:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2
[2017-07-24 14:09:17.231987] I [MSGID: 108026] [afr-self-heal-common.c:1254
:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00
-fd5b-4326-a879-326ff56181de. sources=[0] 1 sinks=2
[2017-07-24 14:09:38.039541] I [MSGID: 108026] [afr-self-heal-common.c:1254
:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2
[2017-07-24 14:09:48.875602] I [MSGID: 108026] [afr-self-heal-common.c:1254
:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00
-fd5b-4326-a879-326ff56181de. sources=[0] 1 sinks=2
[2017-07-24 14:10:39.832068] I [MSGID: 108026] [afr-self-heal-common.c:1254
:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on
e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2
The message "I [MSGID: 108026]
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
sources=[0] 1 sinks=2 " repeated 3 times between [2017-07-24 14:10:
39.832068] and [2017-07-24 14:12:22.686142]
Last message was (I think) because I have reexecute an "heal" command
n.b. dvirtgluster is the RR DNS for all node gluster
> And, what about the selinux?
>
Not sure about this. See if there are disconnect messages in the mount
logs first.
-Ravi
> Thank you
>
>
>
No messages selinux related...
Thank you!
12:42 a.m.
These errors are because not having glusternw assigned to the correct
interface. Once you attach that these errors should go away. This has
nothing to do with the problem you are seeing.
sahina any idea about engine not showing the correct volume info ?
On Mon, Jul 24, 2017 at 7:30 PM, yayo (j) <jaganz(a)gmail.com> wrote:
Hi,
UI refreshed but problem still remain ...
No specific error, I've only these errors but I've read that there is no
problem if I have this kind of errors:
2017-07-24 15:53:59,823+02 INFO [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2)
[b7590c4] START, GlusterServersListVDSCommand(HostName =
node01.localdomain.local, VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 29a62417
2017-07-24 15:54:01,066+02 INFO [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2)
[b7590c4] FINISH, GlusterServersListVDSCommand, return: [10.10.20.80/24:CONNECTED,
node02.localdomain.local:CONNECTED, gdnode04:CONNECTED], log id: 29a62417
2017-07-24 15:54:01,076+02 INFO [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2)
[b7590c4] START, GlusterVolumesListVDSCommand(HostName =
node01.localdomain.local, GlusterVolumesListVDSParameters:{runAsync='true',
hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 7fce25d3
2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode01:/gluster/engine/brick' of volume
'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,212+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode02:/gluster/engine/brick' of volume
'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,215+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode04:/gluster/engine/brick' of volume
'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,218+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode01:/gluster/data/brick' of volume
'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,221+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode02:/gluster/data/brick' of volume
'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,224+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode04:/gluster/data/brick' of volume
'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
2017-07-24 15:54:02,224+02 INFO [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2)
[b7590c4] FINISH, GlusterVolumesListVDSCommand, return: {d19c19e3-910d-437
b-8ba7-4f2a23d17515=org.ovirt.engine.core.common.businessentities.gluste
r.GlusterVolumeEntity@fdc91062, c7a5dfc9-3e72-4ea1-843e-c8275d
4a7c2d=org.ovirt.engine.core.common.businessentities.gluste
r.GlusterVolumeEntity@999a6f23}, log id: 7fce25d3
Thank you
2017-07-24 8:12 GMT+02:00 Kasturi Narra <knarra(a)redhat.com>:
> Hi,
>
> Regarding the UI showing incorrect information about engine and data
> volumes, can you please refresh the UI and see if the issue persists plus
> any errors in the engine.log files ?
>
> Thanks
> kasturi
>
> On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar(a)redhat.com>
> wrote:
>
>>
>> On 07/21/2017 11:41 PM, yayo (j) wrote:
>>
>> Hi,
>>
>> Sorry for follow up again, but, checking the ovirt interface I've found
>> that ovirt report the "engine" volume as an "arbiter"
configuration and the
>> "data" volume as full replicated volume. Check these screenshots:
>>
>>
>> This is probably some refresh bug in the UI, Sahina might be able to
>> tell you.
>>
>>
>> https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFf
>> VmR5aDQ?usp=sharing
>>
>> But the "gluster volume info" command report that all 2 volume are
full
>> replicated:
>>
>>
>> *Volume Name: data*
>> *Type: Replicate*
>> *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d*
>> *Status: Started*
>> *Snapshot Count: 0*
>> *Number of Bricks: 1 x 3 = 3*
>> *Transport-type: tcp*
>> *Bricks:*
>> *Brick1: gdnode01:/gluster/data/brick*
>> *Brick2: gdnode02:/gluster/data/brick*
>> *Brick3: gdnode04:/gluster/data/brick*
>> *Options Reconfigured:*
>> *nfs.disable: on*
>> *performance.readdir-ahead: on*
>> *transport.address-family: inet*
>> *storage.owner-uid: 36*
>> *performance.quick-read: off*
>> *performance.read-ahead: off*
>> *performance.io-cache: off*
>> *performance.stat-prefetch: off*
>> *performance.low-prio-threads: 32*
>> *network.remote-dio: enable*
>> *cluster.eager-lock: enable*
>> *cluster.quorum-type: auto*
>> *cluster.server-quorum-type: server*
>> *cluster.data-self-heal-algorithm: full*
>> *cluster.locking-scheme: granular*
>> *cluster.shd-max-threads: 8*
>> *cluster.shd-wait-qlength: 10000*
>> *features.shard: on*
>> *user.cifs: off*
>> *storage.owner-gid: 36*
>> *features.shard-block-size: 512MB*
>> *network.ping-timeout: 30*
>> *performance.strict-o-direct: on*
>> *cluster.granular-entry-heal: on*
>> *auth.allow: **
>> *server.allow-insecure: on*
>>
>>
>>
>>
>>
>> *Volume Name: engine*
>> *Type: Replicate*
>> *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
>> *Status: Started*
>> *Snapshot Count: 0*
>> *Number of Bricks: 1 x 3 = 3*
>> *Transport-type: tcp*
>> *Bricks:*
>> *Brick1: gdnode01:/gluster/engine/brick*
>> *Brick2: gdnode02:/gluster/engine/brick*
>> *Brick3: gdnode04:/gluster/engine/brick*
>> *Options Reconfigured:*
>> *nfs.disable: on*
>> *performance.readdir-ahead: on*
>> *transport.address-family: inet*
>> *storage.owner-uid: 36*
>> *performance.quick-read: off*
>> *performance.read-ahead: off*
>> *performance.io-cache: off*
>> *performance.stat-prefetch: off*
>> *performance.low-prio-threads: 32*
>> *network.remote-dio: off*
>> *cluster.eager-lock: enable*
>> *cluster.quorum-type: auto*
>> *cluster.server-quorum-type: server*
>> *cluster.data-self-heal-algorithm: full*
>> *cluster.locking-scheme: granular*
>> *cluster.shd-max-threads: 8*
>> *cluster.shd-wait-qlength: 10000*
>> *features.shard: on*
>> *user.cifs: off*
>> *storage.owner-gid: 36*
>> *features.shard-block-size: 512MB*
>> *network.ping-timeout: 30*
>> *performance.strict-o-direct: on*
>> *cluster.granular-entry-heal: on*
>> *auth.allow: **
>>
>> server.allow-insecure: on
>>
>>
>> 2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz(a)gmail.com>:
>>
>>> 2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
>>>
>>>>
>>>> But it does say something. All these gfids of completed heals in the
>>>> log below are the for the ones that you have given the getfattr output
of.
>>>> So what is likely happening is there is an intermittent connection
problem
>>>> between your mount and the brick process, leading to pending heals again
>>>> after the heal gets completed, which is why the numbers are varying each
>>>> time. You would need to check why that is the case.
>>>> Hope this helps,
>>>> Ravi
>>>>
>>>>
>>>>
>>>> *[2017-07-20 09:58:46.573079] I [MSGID: 108026]
>>>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>>>> Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
>>>> sources=[0] 1 sinks=2*
>>>> *[2017-07-20 09:59:22.995003] I [MSGID: 108026]
>>>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>>>> 0-engine-replicate-0: performing metadata selfheal on
>>>> f05b9742-2771-484a-85fc-5b6974bcef81*
>>>> *[2017-07-20 09:59:22.999372] I [MSGID: 108026]
>>>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>>>> Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
>>>> sources=[0] 1 sinks=2*
>>>>
>>>>
>>>
>>> Hi,
>>>
>>> following your suggestion, I've checked the "peer" status and I
found
>>> that there is too many name for the hosts, I don't know if this can be
the
>>> problem or part of it:
>>>
>>> *gluster peer status on NODE01:*
>>> *Number of Peers: 2*
>>>
>>> *Hostname: dnode02.localdomain.local*
>>> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
>>> *State: Peer in Cluster (Connected)*
>>> *Other names:*
>>> *192.168.10.52*
>>> *dnode02.localdomain.local*
>>> *10.10.20.90*
>>> *10.10.10.20*
>>>
>>>
>>>
>>>
>>> *gluster peer status on NODE02:*
>>> *Number of Peers: 2*
>>>
>>> *Hostname: dnode01.localdomain.local*
>>> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
>>> *State: Peer in Cluster (Connected)*
>>> *Other names:*
>>> *gdnode01*
>>> *10.10.10.10*
>>>
>>> *Hostname: gdnode04*
>>> *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828*
>>> *State: Peer in Cluster (Connected)*
>>> *Other names:*
>>> *192.168.10.54*
>>> *10.10.10.40*
>>>
>>>
>>> *gluster peer status on NODE04:*
>>> *Number of Peers: 2*
>>>
>>> *Hostname: dnode02.neridom.dom*
>>> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
>>> *State: Peer in Cluster (Connected)*
>>> *Other names:*
>>> *10.10.20.90*
>>> *gdnode02*
>>> *192.168.10.52*
>>> *10.10.10.20*
>>>
>>> *Hostname: dnode01.localdomain.local*
>>> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
>>> *State: Peer in Cluster (Connected)*
>>> *Other names:*
>>> *gdnode01*
>>> *10.10.10.10*
>>>
>>>
>>>
>>> All these ip are pingable and hosts resolvible across all 3 nodes but,
>>> only the 10.10.10.0 network is the decidated network for gluster (rosolved
>>> using gdnode* host names) ... You think that remove other entries can fix
>>> the problem? So, sorry, but, how can I remove other entries?
>>>
>> I don't think having extra entries could be a problem. Did you check the
>> fuse mount logs for disconnect messages that I referred to in the other
>> email?
>>
>>
>>> And, what about the selinux?
>>>
>> Not sure about this. See if there are disconnect messages in the mount
>> logs first.
>> -Ravi
>>
>>
>>> Thank you
>>>
>>>
>>>
>>
>>
>> --
>> Linux User: 369739 http://counter.li.org
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
--
Linux User: 369739 http://counter.li.org
1:27 a.m.
New subject: [Gluster-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
On Tue, Jul 25, 2017 at 11:12 AM, Kasturi Narra <knarra(a)redhat.com> wrote:
These errors are because not having glusternw assigned to the
correct
interface. Once you attach that these errors should go away. This has
nothing to do with the problem you are seeing.
sahina any idea about engine not showing the correct volume info ?
Please provide the vdsm.log (contianing the gluster volume info) and
engine.log
On Mon, Jul 24, 2017 at 7:30 PM, yayo (j) <jaganz(a)gmail.com>
wrote:
> Hi,
>
> UI refreshed but problem still remain ...
>
> No specific error, I've only these errors but I've read that there is no
> problem if I have this kind of errors:
>
>
> 2017-07-24 15:53:59,823+02 INFO [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2)
> [b7590c4] START, GlusterServersListVDSCommand(HostName =
> node01.localdomain.local, VdsIdVDSCommandParametersBase:{runAsync='true',
> hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 29a62417
> 2017-07-24 15:54:01,066+02 INFO [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2)
> [b7590c4] FINISH, GlusterServersListVDSCommand, return: [10.10.20.80/24:CONNECTED,
> node02.localdomain.local:CONNECTED, gdnode04:CONNECTED], log id: 29a62417
> 2017-07-24 15:54:01,076+02 INFO [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2)
> [b7590c4] START, GlusterVolumesListVDSCommand(HostName =
> node01.localdomain.local, GlusterVolumesListVDSParameters:{runAsync=
> 'true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id:
7fce25d3
> 2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2)
> [b7590c4] Could not associate brick 'gdnode01:/gluster/engine/brick' of
> volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no
> gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
> 2017-07-24 15:54:02,212+02 WARN [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2)
> [b7590c4] Could not associate brick 'gdnode02:/gluster/engine/brick' of
> volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no
> gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
> 2017-07-24 15:54:02,215+02 WARN [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2)
> [b7590c4] Could not associate brick 'gdnode04:/gluster/engine/brick' of
> volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no
> gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
> 2017-07-24 15:54:02,218+02 WARN [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2)
> [b7590c4] Could not associate brick 'gdnode01:/gluster/data/brick' of
> volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no
> gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
> 2017-07-24 15:54:02,221+02 WARN [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2)
> [b7590c4] Could not associate brick 'gdnode02:/gluster/data/brick' of
> volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no
> gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
> 2017-07-24 15:54:02,224+02 WARN [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2)
> [b7590c4] Could not associate brick 'gdnode04:/gluster/data/brick' of
> volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no
> gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
> 2017-07-24 15:54:02,224+02 INFO [org.ovirt.engine.core.vdsbro
> ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2)
> [b7590c4] FINISH, GlusterVolumesListVDSCommand, return: {d19c19e3-910d
> -437b-8ba7-4f2a23d17515=org.ovirt.engine.core.
> common.businessentities.gluster.GlusterVolumeEntity@fdc91062, c7a5dfc9
> -3e72-4ea1-843e-c8275d4a7c2d=org.ovirt.engine.core.c
> ommon.businessentities.gluster.GlusterVolumeEntity@999a6f23}, log id: 7
> fce25d3
>
>
> Thank you
>
>
> 2017-07-24 8:12 GMT+02:00 Kasturi Narra <knarra(a)redhat.com>:
>
>> Hi,
>>
>> Regarding the UI showing incorrect information about engine and data
>> volumes, can you please refresh the UI and see if the issue persists plus
>> any errors in the engine.log files ?
>>
>> Thanks
>> kasturi
>>
>> On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar(a)redhat.com>
>> wrote:
>>
>>>
>>> On 07/21/2017 11:41 PM, yayo (j) wrote:
>>>
>>> Hi,
>>>
>>> Sorry for follow up again, but, checking the ovirt interface I've found
>>> that ovirt report the "engine" volume as an "arbiter"
configuration and the
>>> "data" volume as full replicated volume. Check these screenshots:
>>>
>>>
>>> This is probably some refresh bug in the UI, Sahina might be able to
>>> tell you.
>>>
>>>
>>> https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFf
>>> VmR5aDQ?usp=sharing
>>>
>>> But the "gluster volume info" command report that all 2 volume are
full
>>> replicated:
>>>
>>>
>>> *Volume Name: data*
>>> *Type: Replicate*
>>> *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d*
>>> *Status: Started*
>>> *Snapshot Count: 0*
>>> *Number of Bricks: 1 x 3 = 3*
>>> *Transport-type: tcp*
>>> *Bricks:*
>>> *Brick1: gdnode01:/gluster/data/brick*
>>> *Brick2: gdnode02:/gluster/data/brick*
>>> *Brick3: gdnode04:/gluster/data/brick*
>>> *Options Reconfigured:*
>>> *nfs.disable: on*
>>> *performance.readdir-ahead: on*
>>> *transport.address-family: inet*
>>> *storage.owner-uid: 36*
>>> *performance.quick-read: off*
>>> *performance.read-ahead: off*
>>> *performance.io-cache: off*
>>> *performance.stat-prefetch: off*
>>> *performance.low-prio-threads: 32*
>>> *network.remote-dio: enable*
>>> *cluster.eager-lock: enable*
>>> *cluster.quorum-type: auto*
>>> *cluster.server-quorum-type: server*
>>> *cluster.data-self-heal-algorithm: full*
>>> *cluster.locking-scheme: granular*
>>> *cluster.shd-max-threads: 8*
>>> *cluster.shd-wait-qlength: 10000*
>>> *features.shard: on*
>>> *user.cifs: off*
>>> *storage.owner-gid: 36*
>>> *features.shard-block-size: 512MB*
>>> *network.ping-timeout: 30*
>>> *performance.strict-o-direct: on*
>>> *cluster.granular-entry-heal: on*
>>> *auth.allow: **
>>> *server.allow-insecure: on*
>>>
>>>
>>>
>>>
>>>
>>> *Volume Name: engine*
>>> *Type: Replicate*
>>> *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515*
>>> *Status: Started*
>>> *Snapshot Count: 0*
>>> *Number of Bricks: 1 x 3 = 3*
>>> *Transport-type: tcp*
>>> *Bricks:*
>>> *Brick1: gdnode01:/gluster/engine/brick*
>>> *Brick2: gdnode02:/gluster/engine/brick*
>>> *Brick3: gdnode04:/gluster/engine/brick*
>>> *Options Reconfigured:*
>>> *nfs.disable: on*
>>> *performance.readdir-ahead: on*
>>> *transport.address-family: inet*
>>> *storage.owner-uid: 36*
>>> *performance.quick-read: off*
>>> *performance.read-ahead: off*
>>> *performance.io-cache: off*
>>> *performance.stat-prefetch: off*
>>> *performance.low-prio-threads: 32*
>>> *network.remote-dio: off*
>>> *cluster.eager-lock: enable*
>>> *cluster.quorum-type: auto*
>>> *cluster.server-quorum-type: server*
>>> *cluster.data-self-heal-algorithm: full*
>>> *cluster.locking-scheme: granular*
>>> *cluster.shd-max-threads: 8*
>>> *cluster.shd-wait-qlength: 10000*
>>> *features.shard: on*
>>> *user.cifs: off*
>>> *storage.owner-gid: 36*
>>> *features.shard-block-size: 512MB*
>>> *network.ping-timeout: 30*
>>> *performance.strict-o-direct: on*
>>> *cluster.granular-entry-heal: on*
>>> *auth.allow: **
>>>
>>> server.allow-insecure: on
>>>
>>>
>>> 2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz(a)gmail.com>:
>>>
>>>> 2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar(a)redhat.com>:
>>>>
>>>>>
>>>>> But it does say something. All these gfids of completed heals in
the
>>>>> log below are the for the ones that you have given the getfattr
output of.
>>>>> So what is likely happening is there is an intermittent connection
problem
>>>>> between your mount and the brick process, leading to pending heals
again
>>>>> after the heal gets completed, which is why the numbers are varying
each
>>>>> time. You would need to check why that is the case.
>>>>> Hope this helps,
>>>>> Ravi
>>>>>
>>>>>
>>>>>
>>>>> *[2017-07-20 09:58:46.573079] I [MSGID: 108026]
>>>>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>>>>> Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327.
>>>>> sources=[0] 1 sinks=2*
>>>>> *[2017-07-20 09:59:22.995003] I [MSGID: 108026]
>>>>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>>>>> 0-engine-replicate-0: performing metadata selfheal on
>>>>> f05b9742-2771-484a-85fc-5b6974bcef81*
>>>>> *[2017-07-20 09:59:22.999372] I [MSGID: 108026]
>>>>> [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0:
>>>>> Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81.
>>>>> sources=[0] 1 sinks=2*
>>>>>
>>>>>
>>>>
>>>> Hi,
>>>>
>>>> following your suggestion, I've checked the "peer" status
and I found
>>>> that there is too many name for the hosts, I don't know if this can
be the
>>>> problem or part of it:
>>>>
>>>> *gluster peer status on NODE01:*
>>>> *Number of Peers: 2*
>>>>
>>>> *Hostname: dnode02.localdomain.local*
>>>> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
>>>> *State: Peer in Cluster (Connected)*
>>>> *Other names:*
>>>> *192.168.10.52*
>>>> *dnode02.localdomain.local*
>>>> *10.10.20.90*
>>>> *10.10.10.20*
>>>>
>>>>
>>>>
>>>>
>>>> *gluster peer status on NODE02:*
>>>> *Number of Peers: 2*
>>>>
>>>> *Hostname: dnode01.localdomain.local*
>>>> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
>>>> *State: Peer in Cluster (Connected)*
>>>> *Other names:*
>>>> *gdnode01*
>>>> *10.10.10.10*
>>>>
>>>> *Hostname: gdnode04*
>>>> *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828*
>>>> *State: Peer in Cluster (Connected)*
>>>> *Other names:*
>>>> *192.168.10.54*
>>>> *10.10.10.40*
>>>>
>>>>
>>>> *gluster peer status on NODE04:*
>>>> *Number of Peers: 2*
>>>>
>>>> *Hostname: dnode02.neridom.dom*
>>>> *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd*
>>>> *State: Peer in Cluster (Connected)*
>>>> *Other names:*
>>>> *10.10.20.90*
>>>> *gdnode02*
>>>> *192.168.10.52*
>>>> *10.10.10.20*
>>>>
>>>> *Hostname: dnode01.localdomain.local*
>>>> *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12*
>>>> *State: Peer in Cluster (Connected)*
>>>> *Other names:*
>>>> *gdnode01*
>>>> *10.10.10.10*
>>>>
>>>>
>>>>
>>>> All these ip are pingable and hosts resolvible across all 3 nodes but,
>>>> only the 10.10.10.0 network is the decidated network for gluster
(rosolved
>>>> using gdnode* host names) ... You think that remove other entries can
fix
>>>> the problem? So, sorry, but, how can I remove other entries?
>>>>
>>> I don't think having extra entries could be a problem. Did you check
>>> the fuse mount logs for disconnect messages that I referred to in the other
>>> email?
>>>
>>>
>>>> And, what about the selinux?
>>>>
>>> Not sure about this. See if there are disconnect messages in the mount
>>> logs first.
>>> -Ravi
>>>
>>>
>>>> Thank you
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Linux User: 369739 http://counter.li.org
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>
>
>
> --
> Linux User: 369739 http://counter.li.org
>
_______________________________________________
Gluster-users mailing list
Gluster-users(a)gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
3:15 a.m.
2017-07-25 7:42 GMT+02:00 Kasturi Narra <knarra(a)redhat.com>:
These errors are because not having glusternw assigned to the
correct
interface. Once you attach that these errors should go away. This has
nothing to do with the problem you are seeing.
Hi,
You talking about errors like these?
2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode01:/gluster/engine/brick' of volume
'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
How to assign "glusternw (???)" to the correct interface?
Other errors on unsync gluster elements still remain... This is a
production env, so, there is any chance to subscribe to RH support?
Thank you
4:31 a.m.
New subject: [Gluster-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
On Tue, Jul 25, 2017 at 1:45 PM, yayo (j) <jaganz(a)gmail.com> wrote:
2017-07-25 7:42 GMT+02:00 Kasturi Narra <knarra(a)redhat.com>:
> These errors are because not having glusternw assigned to the correct
> interface. Once you attach that these errors should go away. This has
> nothing to do with the problem you are seeing.
>
Hi,
You talking about errors like these?
2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro
ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4]
Could not associate brick 'gdnode01:/gluster/engine/brick' of volume
'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster
network found in cluster '00000002-0002-0002-0002-00000000017a'
How to assign "glusternw (???)" to the correct interface?
https://www.ovirt.org/blog/2017/04/up-and-running-with-ovirt-4.1-and-glus...
"Storage network" section explains this. Please make sure that gdnode01 is
resolvable from engine.
Other errors on unsync gluster elements still remain... This is a
production env, so, there is any chance to subscribe to RH support?
The unsynced entries - did you check for disconnect messages in the mount
log as suggested by Ravi?
For Red Hat support, the best option is to contact your local Red Hat
representative.
Thank you
_______________________________________________
Gluster-users mailing list
Gluster-users(a)gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
9:58 a.m.
New subject: [Gluster-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
2017-07-25 11:31 GMT+02:00 Sahina Bose <sabose(a)redhat.com>:
> Other errors on unsync gluster elements still remain... This is a
> production env, so, there is any chance to subscribe to RH support?
>
The unsynced entries - did you check for disconnect messages in the mount
log as suggested by Ravi?
Hi have provided this (check past mails): * tail -f
/var/log/glusterfs/rhev-data-center-mnt-glusterSD-dvirtgluster\:engine.log*
Is enougth?
Thank you
4:10 a.m.
2017-07-19 11:22 GMT+02:00 yayo (j) <jaganz(a)gmail.com>:
running the "gluster volume heal engine" don't solve
the problem...
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1
arbiter to 3 full replicated cluster but i don't know this is the problem...
Hi,
I'm sorry for the follow up. I want to say that after upgrade all nodes to
the same level, all problems are solved and cluster works very well now!
Thank you all for the support!
2429
days inactive
2480
days old
20 comments
4 participants
participants (4)
-
Kasturi Narra -
Ravishankar N -
Sahina Bose -
yayo (j)