ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements
Hi all, We have an ovirt cluster hyperconverged with hosted engine on 3 full replicated node . This cluster have 2 gluster volume: - data: volume for the Data (Master) Domain (For vm) - engine: volume fro the hosted_storage Domain (for hosted engine) We have this problem: "engine" gluster volume have always unsynced elements and we cant' fix the problem, on command line we have tried to use the "heal" command but elements remain always unsynced .... Below the heal command "status": [root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20 /__DIRECT_IO_TEST__ Status: Connected Number of entries: 12 Brick node02:/gluster/engine/brick /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01 <gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f> /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids <gfid:1e309376-c62e-424f-9857-f9a0c3a729bf> <gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9> <gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a> /__DIRECT_IO_TEST__ <gfid:67606789-1f34-4c15-86b8-c0d05b07f187> <gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0> /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6 <gfid:9ad720b2-507d-4830-8294-ec8adee6d384> <gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf> Status: Connected Number of entries: 12 Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0 running the "gluster volume heal engine" don't solve the problem... Some extra info: We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster but i don't know this is the problem... The "data" volume is good and healty and have no unsynced entry. Ovirt refuse to put the node02 and node01 in "maintenance mode" and complains about "unsynced elements" How can I fix this? Thank you
[Adding gluster-users] On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) <jaganz@gmail.com> wrote:
Hi all,
We have an ovirt cluster hyperconverged with hosted engine on 3 full replicated node . This cluster have 2 gluster volume:
- data: volume for the Data (Master) Domain (For vm) - engine: volume fro the hosted_storage Domain (for hosted engine)
We have this problem: "engine" gluster volume have always unsynced elements and we cant' fix the problem, on command line we have tried to use the "heal" command but elements remain always unsynced ....
Below the heal command "status":
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267- 52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053- a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20 /__DIRECT_IO_TEST__ Status: Connected Number of entries: 12
Brick node02:/gluster/engine/brick /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267- 52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01 <gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f> /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids <gfid:1e309376-c62e-424f-9857-f9a0c3a729bf> <gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9> <gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a> /__DIRECT_IO_TEST__ <gfid:67606789-1f34-4c15-86b8-c0d05b07f187> <gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0> /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053- a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6 <gfid:9ad720b2-507d-4830-8294-ec8adee6d384> <gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf> Status: Connected Number of entries: 12
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
running the "gluster volume heal engine" don't solve the problem...
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster but i don't know this is the problem...
The "data" volume is good and healty and have no unsynced entry.
Ovirt refuse to put the node02 and node01 in "maintenance mode" and complains about "unsynced elements"
How can I fix this? Thank you
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a multi-part message in MIME format. --------------0C293E7D7C0723BAFB7F6D00 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/19/2017 08:02 PM, Sahina Bose wrote:
[Adding gluster-users]
On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) <jaganz@gmail.com <mailto:jaganz@gmail.com>> wrote:
Hi all,
We have an ovirt cluster hyperconverged with hosted engine on 3 full replicated node . This cluster have 2 gluster volume:
- data: volume for the Data (Master) Domain (For vm) - engine: volume fro the hosted_storage Domain (for hosted engine)
We have this problem: "engine" gluster volume have always unsynced elements and we cant' fix the problem, on command line we have tried to use the "heal" command but elements remain always unsynced ....
Below the heal command "status":
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61 /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1 /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20 /__DIRECT_IO_TEST__ Status: Connected Number of entries: 12
Brick node02:/gluster/engine/brick /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01 <gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f> /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids <gfid:1e309376-c62e-424f-9857-f9a0c3a729bf> <gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9> <gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a> /__DIRECT_IO_TEST__ <gfid:67606789-1f34-4c15-86b8-c0d05b07f187> <gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0> /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6 <gfid:9ad720b2-507d-4830-8294-ec8adee6d384> <gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf> Status: Connected Number of entries: 12
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
running the "gluster volume heal engine" don't solve the problem...
1. What does the glustershd.log say on all 3 nodes when you run the command? Does it complain anything about these files? 2. Are these 12 files also present in the 3rd data brick? 3. Can you provide the output of `gluster volume info` for the this volume?
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3? Thanks, Ravi
but i don't know this is the problem...
The "data" volume is good and healty and have no unsynced entry.
Ovirt refuse to put the node02 and node01 in "maintenance mode" and complains about "unsynced elements"
How can I fix this? Thank you
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
--------------0C293E7D7C0723BAFB7F6D00 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p><br> </p> <br> <div class="moz-cite-prefix">On 07/19/2017 08:02 PM, Sahina Bose wrote:<br> </div> <blockquote type="cite" cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com"> <div dir="ltr">[Adding gluster-users]<br> </div> <div class="gmail_extra"><br> <div class="gmail_quote">On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) <span dir="ltr"><<a href="mailto:jaganz@gmail.com" target="_blank" moz-do-not-send="true">jaganz@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr">Hi all, <div><br> </div> <div>We have an ovirt cluster hyperconverged with hosted engine on 3 full replicated node . This cluster have 2 gluster volume:</div> <div><br> </div> <div>- data: volume for the Data (Master) Domain (For vm)</div> <div>- engine: volume fro the hosted_storage Domain (for hosted engine)</div> <div><br> </div> <div>We have this problem: "engine" gluster volume have always unsynced elements and we cant' fix the problem, on command line we have tried to use the "heal" command but elements remain always unsynced .... </div> <div><br> </div> <div>Below the heal command "status":</div> <div><br> </div> <div> <div class="m_6699510126617823271gmail-hljs m_6699510126617823271gmail-objectivec" style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace"> <div>[root@node01 ~]<span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"># gluster volume heal engine info</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Brick node01:/gluster/engine/brick</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.48</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.64</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.60</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.2</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.68</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/19d71267-<wbr>52a4-42a3-bb1e-e3145361c0c2/<wbr>7a215635-02f3-47db-80db-<wbr>8b689c6a8f01</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/88d41053-<wbr>a257-4272-9e2e-2f3de0743b81/<wbr>6573ed08-d3ed-4d12-9227-<wbr>2c95941e1ad6</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.61</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.1</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/dom_md/ids</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.20</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/__DIRECT_IO_TEST__</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Status: Connected</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Number of entries: 12</span></div> <div><br> </div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Brick node02:/gluster/engine/brick</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/19d71267-<wbr>52a4-42a3-bb1e-e3145361c0c2/<wbr>7a215635-02f3-47db-80db-<wbr>8b689c6a8f01</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:9a601373-bbaa-44d8-b396-<wbr>f0b9b12c026f></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/dom_md/ids</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:1e309376-c62e-424f-9857-<wbr>f9a0c3a729bf></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:e3565b50-1495-4e5b-ae88-<wbr>3bceca47b7d9></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:4e33ac33-dddb-4e29-b4a3-<wbr>51770b81166a></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/__DIRECT_IO_TEST__</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:67606789-1f34-4c15-86b8-<wbr>c0d05b07f187></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:9ef88647-cfe6-4a35-a38c-<wbr>a5173c9e8fc0></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/88d41053-<wbr>a257-4272-9e2e-2f3de0743b81/<wbr>6573ed08-d3ed-4d12-9227-<wbr>2c95941e1ad6</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:9ad720b2-507d-4830-8294-<wbr>ec8adee6d384></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)"><span class="m_6699510126617823271gmail-hljs-meta-string" style="color:rgb(77,153,191)"><gfid:d9853e5d-a2bf-4cee-8b39-<wbr>7781a98033cf></span></span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Status: Connected</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Number of entries: 12</span></div> <div><br> </div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Brick node04:/gluster/engine/brick</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Status: Connected</span></div> <div><span class="m_6699510126617823271gmail-hljs-meta" style="color:rgb(31,113,153)">Number of entries: 0</span></div> <div><br> </div> </div> </div> <br> <div> </div> <div>running the "gluster volume heal engine" don't solve the problem...<br> </div> <div><br> </div> </div> </blockquote> </div> </div> </blockquote> <br> 1. What does the glustershd.log say on all 3 nodes when you run the command? Does it complain anything about these files?<br> 2. Are these 12 files also present in the 3rd data brick?<br> 3. Can you provide the output of `gluster volume info` for the this volume?<br> <blockquote type="cite" cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div>Some extra info:</div> <div><br> </div> <div>We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster </div> </div> </blockquote> </div> </div> </blockquote> <br> Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?<br> <br> Thanks,<br> Ravi<br> <blockquote type="cite" cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div>but i don't know this is the problem...</div> <div><br> </div> <div>The "data" volume is good and healty and have no unsynced entry.</div> <div><br> </div> <div>Ovirt refuse to put the node02 and node01 in "maintenance mode" and complains about "unsynced elements"<br> </div> <div><br> </div> <div>How can I fix this?</div> <div>Thank you </div> </div> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a href="mailto:Users@ovirt.org" moz-do-not-send="true">Users@ovirt.org</a><br> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br> <br> </blockquote> </div> <br> </div> </blockquote> <br> </body> </html> --------------0C293E7D7C0723BAFB7F6D00--
Hi, Thank you for the answer and sorry for delay: 2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishankar@redhat.com>: 1. What does the glustershd.log say on all 3 nodes when you run the
command? Does it complain anything about these files?
No, glustershd.log is clean, no extra log after command on all 3 nodes
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this volume?
*Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: node01:/gluster/engine/brick* *Brick2: node02:/gluster/engine/brick* *Brick3: node04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** server.allow-insecure: on
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?
Yes #gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force *(OK!)* #gluster volume heal engine info *(no entries!)* #gluster volume add-brick engine replica 3 node04:/gluster/engine/brick *(OK!)* *After some minutes* [root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick Status: Connected Number of entries: 0 Brick node02:/gluster/engine/brick Status: Connected Number of entries: 0 Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
Thanks, Ravi
Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out Thank you!
This is a multi-part message in MIME format. --------------C563F370DBD059BBFD94E893 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/20/2017 02:20 PM, yayo (j) wrote:
Hi,
Thank you for the answer and sorry for delay:
2017-07-19 16:55 GMT+02:00 Ravishankar N <ravishankar@redhat.com <mailto:ravishankar@redhat.com>>:
1. What does the glustershd.log say on all 3 nodes when you run the command? Does it complain anything about these files?
No, glustershd.log is clean, no extra log after command on all 3 nodes
Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that. If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen. If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex //gluster/engine/brick//path-to-file` ? Thanks, Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this volume?
/Volume Name: engine/ /Type: Replicate/ /Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/ /Status: Started/ /Snapshot Count: 0/ /Number of Bricks: 1 x 3 = 3/ /Transport-type: tcp/ /Bricks:/ /Brick1: node01:/gluster/engine/brick/ /Brick2: node02:/gluster/engine/brick/ /Brick3: node04:/gluster/engine/brick/ /Options Reconfigured:/ /nfs.disable: on/ /performance.readdir-ahead: on/ /transport.address-family: inet/ /storage.owner-uid: 36/ /performance.quick-read: off/ /performance.read-ahead: off/ /performance.io-cache: off/ /performance.stat-prefetch: off/ /performance.low-prio-threads: 32/ /network.remote-dio: off/ /cluster.eager-lock: enable/ /cluster.quorum-type: auto/ /cluster.server-quorum-type: server/ /cluster.data-self-heal-algorithm: full/ /cluster.locking-scheme: granular/ /cluster.shd-max-threads: 8/ /cluster.shd-wait-qlength: 10000/ /features.shard: on/ /user.cifs: off/ /storage.owner-gid: 36/ /features.shard-block-size: 512MB/ /network.ping-timeout: 30/ /performance.strict-o-direct: on/ /cluster.granular-entry-heal: on/ /auth.allow: */
server.allow-insecure: on
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node02:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
Thanks, Ravi
Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out
Thank you!
--------------C563F370DBD059BBFD94E893 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 07/20/2017 02:20 PM, yayo (j) wrote:<br> </div> <blockquote type="cite" cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">Hi, </div> <div class="gmail_quote"><br> </div> <div class="gmail_quote">Thank you for the answer and sorry for delay:</div> <div class="gmail_quote"><br> </div> <div class="gmail_quote">2017-07-19 16:55 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>></span>:</div> <div class="gmail_quote"><br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 1. What does the glustershd.log say on all 3 nodes when you run the command? Does it complain anything about these files?<br> </div> </blockquote> <div><br> </div> <div>No, glustershd.log is clean, no extra log after command on all 3 nodes</div> </div> </div> </div> </blockquote> <br> Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that.<br> If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.<br> <br> If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex <i>/gluster/engine/brick/</i>path-to-file` ?<br> <br> Thanks,<br> Ravi<br> <br> <blockquote type="cite" cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 2. Are these 12 files also present in the 3rd data brick?<br> </div> </blockquote> <div><br> </div> <div>I've checked right now: all files exists in all 3 nodes </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 3. Can you provide the output of `gluster volume info` for the this volume?</div> </blockquote> <div><br> </div> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume Name: engine</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Type: Replicate</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Status: Started</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Snapshot Count: 0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Bricks: 1 x 3 = 3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Transport-type: tcp</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Bricks:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick1: node01:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick2: node02:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick3: node04:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Options Reconfigured:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>nfs.disable: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.readdir-ahead: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>transport.address-family: inet</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-uid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.quick-read: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.read-ahead: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.io-cache: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.stat-prefetch: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.low-prio-threads: 32</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.remote-dio: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.eager-lock: enable</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.quorum-type: auto</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.server-quorum-type: server</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.data-self-heal-algorithm: full</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.locking-scheme: granular</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-max-threads: 8</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-wait-qlength: 10000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>user.cifs: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-gid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard-block-size: 512MB</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.ping-timeout: 30</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.strict-o-direct: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.granular-entry-heal: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>auth.allow: *</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div> server.allow-insecure: on</div> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="gmail-"><br> <blockquote type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div>Some extra info:</div> <div><br> </div> <div>We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster </div> </div> </blockquote> </div> </div> </blockquote> <br> </span> Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?<br> <br> </div> </blockquote> <div><br> </div> <div>Yes</div> <br> <div> <div class="gmail-hljs gmail-nginx" style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace"> <div><br> </div> <div> <div><span class="gmail-hljs-attribute">#gluster</span> volume remove-brick engine replica <span class="gmail-hljs-number" style="color:rgb(136,0,0)">2</span> node03:/gluster/data/brick force <b>(OK!)</b></div> <div><br> </div> <div>#gluster volume heal engine <span class="gmail-hljs-literal" style="color:rgb(120,169,96)">info</span> <b>(<span class="gmail-hljs-literal" style="color:rgb(120,169,96)">no</span> entries!)</b></div> <div><br> </div> <div>#gluster volume add-brick engine replica <span class="gmail-hljs-number" style="color:rgb(136,0,0)">3</span> node04:/gluster/engine/brick <b>(OK!)</b></div> <div><br> </div> <div><b>After some minutes</b></div> <div><br> </div> <div>[root<span class="gmail-hljs-variable" style="color:rgb(188,96,96)">@node01</span> ~]<span class="gmail-hljs-comment" style="color:rgb(136,136,136)"># gluster volume heal engine info</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node01:/gluster/engine/brick</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node02:/gluster/engine/brick</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node04:/gluster/engine/brick</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> </div> </div> </div> <br> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> Thanks,<br> Ravi</div> </blockquote> <div><br> </div> <div>Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out </div> <div><br> </div> <div>Thank you!</div> </div> <div><br> </div> </div> </div> </blockquote> <br> </body> </html> --------------C563F370DBD059BBFD94E893--
2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that. If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.
I've executed the command on all 3 nodes (Know is enougth only one) , after that the "heal" command report elements between 6 and 10 ... (sometime 6, sometime 8, sometime 10) Log on glustershd.log don't say anything : *[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex */gluster/engine/brick/* path-to-file` ?
*NODE01:* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000120000000000000000* *trusted.bit-rot.version=0x090000000000000059647d5b000447e9* *trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x0000000e0000000000000000* *trusted.bit-rot.version=0x090000000000000059647d5b000447e9* *trusted.gfid=0x676067891f344c1586b8c0d05b07f187* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000550000000000000000* *trusted.bit-rot.version=0x090000000000000059647d5b000447e9* *trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000070000000000000000* *trusted.bit-rot.version=0x090000000000000059647d5b000447e9* *trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000000000000000000000* *trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9* *trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/__DIRECT_IO_TEST__* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000000000000000000000* *trusted.gfid=0xf05b97422771484a85fc5b6974bcef81* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000010000000000000000* *trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9* *trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x0000000a0000000000000000* *trusted.bit-rot.version=0x090000000000000059647d5b000447e9* *trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0* *NODE02:* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x0000001a0000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48* *trusted.afr.dirty=0x000000010000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x0000000c0000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0x676067891f344c1586b8c0d05b07f187* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-1=0x000000000000000000000000* *trusted.afr.engine-client-2=0x0000008e0000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000090000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000010000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/__DIRECT_IO_TEST__* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000000000000000000000* *trusted.gfid=0xf05b97422771484a85fc5b6974bcef81* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000020000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64* *trusted.afr.dirty=0x000000000000000000000000* *trusted.afr.engine-client-0=0x000000000000000000000000* *trusted.afr.engine-client-2=0x000000120000000000000000* *trusted.bit-rot.version=0x08000000000000005965ede0000c352d* *trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0* *NODE04:* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0x676067891f344c1586b8c0d05b07f187* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.afr.dirty=0x000000000000000000000000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/__DIRECT_IO_TEST__* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.bit-rot.version=0x0200000000000000596484e20006237b* *trusted.gfid=0xf05b97422771484a85fc5b6974bcef81* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.afr.dirty=0x000000000000000000000000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327* *trusted.glusterfs.shard.block-size=0x0000000020000000* *trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000* *getfattr: Removing leading '/' from absolute path names* *# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64* *security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000* *trusted.bit-rot.version=0x050000000000000059662c390006b836* *trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0* hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how? *[root@node01 ~]# sestatus* *SELinux status: disabled* *[root@node02 ~]# sestatus* *SELinux status: disabled* *[root@node04 ~]# sestatus* *SELinux status: disabled* Thank you
Thanks, Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this volume?
*Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: node01:/gluster/engine/brick* *Brick2: node02:/gluster/engine/brick* *Brick3: node04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: **
server.allow-insecure: on
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node02:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
Thanks, Ravi
Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out
Thank you!
-- Linux User: 369739 http://counter.li.org
This is a multi-part message in MIME format. --------------5903BBB64769515A20658214 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/20/2017 03:42 PM, yayo (j) wrote:
2017-07-20 11:34 GMT+02:00 Ravishankar N <ravishankar@redhat.com <mailto:ravishankar@redhat.com>>:
Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that. If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.
I've executed the command on all 3 nodes (Know is enougth only one) , after that the "heal" command report elements between 6 and 10 ... (sometime 6, sometime 8, sometime 10)
Log on glustershd.log don't say anything :
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
/[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/ /[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81/ /[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex //gluster/engine/brick//path-to-file` ?
*/NODE01:/* /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000120000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000000e0000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x676067891f344c1586b8c0d05b07f187/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000550000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000070000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000000000000000000000/ /trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9/ /trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/__DIRECT_IO_TEST__/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000000000000000000000/ /trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000010000000000000000/ /trusted.bit-rot.version=0x0f0000000000000059647d5b000447e9/ /trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000000a0000000000000000/ /trusted.bit-rot.version=0x090000000000000059647d5b000447e9/ /trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/ / / */ /* */NODE02:/* /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000001a0000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/ /trusted.afr.dirty=0x000000010000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000000c0000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x676067891f344c1586b8c0d05b07f187/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-1=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x0000008e0000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000090000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000010000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/__DIRECT_IO_TEST__/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000000000000000000000/ /trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000020000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.afr.engine-client-0=0x000000000000000000000000/ /trusted.afr.engine-client-2=0x000000120000000000000000/ /trusted.bit-rot.version=0x08000000000000005965ede0000c352d/ /trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/ / / / / / / / / /*NODE04*:/ /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0xe3565b5014954e5bae883bceca47b7d9/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x676067891f344c1586b8c0d05b07f187/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x8aa745646740403ead51f56d9ca5d7a7/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000c8000000000000000000000000000000000d4f2290000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x4e33ac33dddb4e29b4a351770b81166a/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x2581cb9ac2b74bd9ac17a09bd2f001b3/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/__DIRECT_IO_TEST__/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x0200000000000000596484e20006237b/ /trusted.gfid=0xf05b97422771484a85fc5b6974bcef81/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000000000000000000000000000000000000000000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.afr.dirty=0x000000000000000000000000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0xe6dfd556340b4b76b47b7b6f5bd74327/ /trusted.glusterfs.shard.block-size=0x0000000020000000/ /trusted.glusterfs.shard.file-size=0x0000000000100000000000000000000000000000000008000000000000000000/ / / /getfattr: Removing leading '/' from absolute path names/ /# file: gluster/engine/brick/.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64/ /security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000/ /trusted.bit-rot.version=0x050000000000000059662c390006b836/ /trusted.gfid=0x9ef88647cfe64a35a38ca5173c9e8fc0/
hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how?
/[root@node01 ~]# sestatus/ /SELinux status: disabled/ / / /[root@node02 ~]# sestatus/ /SELinux status: disabled/ / / /[root@node04 ~]# sestatus/ /SELinux status: disabled/
Thank you
Thanks, Ravi
2. Are these 12 files also present in the 3rd data brick?
I've checked right now: all files exists in all 3 nodes
3. Can you provide the output of `gluster volume info` for the this volume?
/Volume Name: engine/ /Type: Replicate/ /Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/ /Status: Started/ /Snapshot Count: 0/ /Number of Bricks: 1 x 3 = 3/ /Transport-type: tcp/ /Bricks:/ /Brick1: node01:/gluster/engine/brick/ /Brick2: node02:/gluster/engine/brick/ /Brick3: node04:/gluster/engine/brick/ /Options Reconfigured:/ /nfs.disable: on/ /performance.readdir-ahead: on/ /transport.address-family: inet/ /storage.owner-uid: 36/ /performance.quick-read: off/ /performance.read-ahead: off/ /performance.io-cache: off/ /performance.stat-prefetch: off/ /performance.low-prio-threads: 32/ /network.remote-dio: off/ /cluster.eager-lock: enable/ /cluster.quorum-type: auto/ /cluster.server-quorum-type: server/ /cluster.data-self-heal-algorithm: full/ /cluster.locking-scheme: granular/ /cluster.shd-max-threads: 8/ /cluster.shd-wait-qlength: 10000/ /features.shard: on/ /user.cifs: off/ /storage.owner-gid: 36/ /features.shard-block-size: 512MB/ /network.ping-timeout: 30/ /performance.strict-o-direct: on/ /cluster.granular-entry-heal: on/ /auth.allow: */
server.allow-insecure: on
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster
Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?
Yes
#gluster volume remove-brick engine replica 2 node03:/gluster/data/brick force *(OK!)*
#gluster volume heal engine info *(no entries!)*
#gluster volume add-brick engine replica 3 node04:/gluster/engine/brick *(OK!)*
*After some minutes*
[root@node01 ~]# gluster volume heal engine info Brick node01:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node02:/gluster/engine/brick Status: Connected Number of entries: 0
Brick node04:/gluster/engine/brick Status: Connected Number of entries: 0
Thanks, Ravi
Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out
Thank you!
-- Linux User: 369739 http://counter.li.org
--------------5903BBB64769515A20658214 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p><br> </p> <br> <div class="moz-cite-prefix">On 07/20/2017 03:42 PM, yayo (j) wrote:<br> </div> <blockquote type="cite" cite="mid:CAGK=3kycgTSLxRLZSPNZi+ERh_=OhDFH29hXHzWEPnsV3vmhAQ@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"><br> <div class="gmail_quote">2017-07-20 11:34 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="gmail-m_7883186325552308780gmail-"> <br> <div class="gmail-m_7883186325552308780gmail-m_9132661499198415129moz-cite-prefix">Could you check if the self-heal daemon on all nodes is connected to the 3 bricks? You will need to check the glustershd.log for that.<br> </div> </span> If it is not connected, try restarting the shd using `gluster volume start engine force`, then launch the heal command like you did earlier and see if heals happen.<br> <br> </div> </blockquote> <div><br> </div> <div>I've executed the command on all 3 nodes (Know is enougth only one) , after that the "heal" command report elements between 6 and 10 ... (sometime 6, sometime 8, sometime 10)</div> <div><br> </div> <div><br> </div> <div>Log on glustershd.log don't say anything :</div> </div> </div> </div> </blockquote> <br> But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case.<br> Hope this helps,<br> Ravi<br> <br> <blockquote type="cite" cite="mid:CAGK=3kycgTSLxRLZSPNZi+ERh_=OhDFH29hXHzWEPnsV3vmhAQ@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:<wbr>afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-<wbr>7b6f5bd74327. sources=[0] 1 sinks=2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-<wbr>5b6974bcef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:<wbr>afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-<wbr>5b6974bcef81. sources=[0] 1 sinks=2</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> If it doesn't, please provide the getfattr outputs of the 12 files from all 3 nodes using `getfattr -d -m . -e hex <i>/gluster/engine/brick/</i>path-to-<wbr>file` ?<br> <br> </div> </blockquote> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><b><i>NODE01:</i></b></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000120000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000000e0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000550000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000070000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x0f0000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000010000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x0f0000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000000a0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x090000000000000059647d5b0004<wbr>47e9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><b><i><br> </i></b></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><b><i>NODE02:</i></b></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000001a0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000010000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000000c0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-1=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x0000008e0000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000090000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000010000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000020000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-0=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.engine-client-2=<wbr>0x000000120000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x08000000000000005965ede0000c<wbr>352d</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><b>NODE04</b>:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.68</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe3565b5014954e5bae883bceca47<wbr>b7d9</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.48</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x676067891f344c1586b8c0d05b07<wbr>f187</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/19d71267-52a4-42a3-<wbr>bb1e-e3145361c0c2/7a215635-<wbr>02f3-47db-80db-8b689c6a8f01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x8aa745646740403ead51f56d9ca5<wbr>d7a7</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000c80000000000000000000<wbr>00000000000000d4f2290000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.60</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x4e33ac33dddb4e29b4a351770b81<wbr>166a</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>dom_md/ids</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x2581cb9ac2b74bd9ac17a09bd2f0<wbr>01b3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/__DIRECT_<wbr>IO_TEST__</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x0200000000000000596484e20006<wbr>237b</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xf05b97422771484a85fc5b6974bc<wbr>ef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000000000000000000000<wbr>000000000000000000000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/8f215dd2-<wbr>8531-4a4f-b6ed-ea789dd8821b/<wbr>images/88d41053-a257-4272-<wbr>9e2e-2f3de0743b81/6573ed08-<wbr>d3ed-4d12-9227-2c95941e1ad6</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.afr.dirty=<wbr>0x000000000000000000000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0xe6dfd556340b4b76b47b7b6f5bd7<wbr>4327</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.block-<wbr>size=0x0000000020000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.glusterfs.shard.file-<wbr>size=<wbr>0x0000000000100000000000000000<wbr>000000000000000008000000000000<wbr>000000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>getfattr: Removing leading '/' from absolute path names</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i># file: gluster/engine/brick/.shard/<wbr>8aa74564-6740-403e-ad51-<wbr>f56d9ca5d7a7.64</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.bit-rot.version=<wbr>0x050000000000000059662c390006<wbr>b836</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>trusted.gfid=<wbr>0x9ef88647cfe64a35a38ca5173c9e<wbr>8fc0</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div>hum.... Is selinux the problem? but on node04 was disabled (AFTER GLUSTER JOIN, I hope to remember) ... You think I needs to relabel? how?</div> <div><br> </div> </div> </div> <blockquote style="margin:0 0 0 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[root@node01 ~]# sestatus</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>SELinux status: disabled</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[root@node02 ~]# sestatus</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>SELinux status: disabled</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div><i><br> </i></div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[root@node04 ~]# sestatus</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>SELinux status: disabled</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div><br> </div> <div>Thank you </div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> Thanks,<br> Ravi <div> <div class="gmail-m_7883186325552308780gmail-h5"><br> <br> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 2. Are these 12 files also present in the 3rd data brick?<br> </div> </blockquote> <div><br> </div> <div>I've checked right now: all files exists in all 3 nodes </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> 3. Can you provide the output of `gluster volume info` for the this volume?</div> </blockquote> <div><br> </div> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume Name: engine</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Type: Replicate</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Volume ID: d19c19e3-910d-437b-8ba7-4f2a23<wbr>d17515</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Status: Started</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Snapshot Count: 0</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Bricks: 1 x 3 = 3</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Transport-type: tcp</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Bricks:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick1: node01:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick2: node02:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Brick3: node04:/gluster/engine/brick</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Options Reconfigured:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>nfs.disable: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.readdir-ahead: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>transport.address-family: inet</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-uid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.quick-read: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.read-ahead: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.io-cache: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.stat-prefetch: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.low-prio-threads: 32</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.remote-dio: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.eager-lock: enable</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.quorum-type: auto</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.server-quorum-type: server</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.data-self-heal-algorit<wbr>hm: full</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.locking-scheme: granular</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-max-threads: 8</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.shd-wait-qlength: 10000</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>user.cifs: off</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>storage.owner-gid: 36</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>features.shard-block-size: 512MB</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>network.ping-timeout: 30</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>performance.strict-o-direct: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>cluster.granular-entry-heal: on</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>auth.allow: *</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div> server.allow-insecure: on</div> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-"><br> <blockquote type="cite"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div>Some extra info:</div> <div><br> </div> <div>We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster </div> </div> </blockquote> </div> </div> </blockquote> <br> </span> Just curious, how did you do this? `remove-brick` of arbiter brick followed by an `add-brick` to increase to replica-3?<br> <br> </div> </blockquote> <div><br> </div> <div>Yes</div> <br> <div> <div class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-nginx" style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace"> <div><br> </div> <div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-attribute">#gluster</span> volume remove-brick engine replica <span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-number" style="color:rgb(136,0,0)">2</span> node03:/gluster/data/brick force <b>(OK!)</b></div> <div><br> </div> <div>#gluster volume heal engine <span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-literal" style="color:rgb(120,169,96)">info</span> <b>(<span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-literal" style="color:rgb(120,169,96)">no</span> entries!)</b></div> <div><br> </div> <div>#gluster volume add-brick engine replica <span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-number" style="color:rgb(136,0,0)">3</span> node04:/gluster/engine/brick <b>(OK!)</b></div> <div><br> </div> <div><b>After some minutes</b></div> <div><br> </div> <div>[root<span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-variable" style="color:rgb(188,96,96)">@node01</span> ~]<span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)"># gluster volume heal engine info</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node01:/gluster/engine/brick</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node02:/gluster/engine/brick</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> <div><br> </div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node04:/gluster/engine/brick</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div> <div><span class="gmail-m_7883186325552308780gmail-m_9132661499198415129gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div> </div> </div> </div> <br> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"> Thanks,<br> Ravi</div> </blockquote> <div><br> </div> <div>Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out </div> <div><br> </div> <div>Thank you!</div> </div> <div><br> </div> </div> </div> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> <br clear="all"> <div><br> </div> -- <br> <div class="gmail-m_7883186325552308780gmail_signature">Linux User: 369739 <a href="http://counter.li.org" target="_blank" moz-do-not-send="true">http://counter.li.org</a></div> </div> </div> </blockquote> <br> </body> </html> --------------5903BBB64769515A20658214--
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi, But we ha1e 2 gluster volume on the same network and the other one (the "Data" gluster) don't have any problems. Why you think there is a network problem? How to check this on a gluster infrastructure? Thank you
This is a multi-part message in MIME format. --------------84EB6DEB84AAFAA57D0538D6 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/21/2017 02:55 PM, yayo (j) wrote:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com <mailto:ravishankar@redhat.com>>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
/[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/ /[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81/ /[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
Hi,
But we ha1e 2 gluster volume on the same network and the other one (the "Data" gluster) don't have any problems. Why you think there is a network problem?
Because pending self-heals come into the picture when I/O from the clients (mounts) do not succeed on some bricks. They are mostly due to (a) the client losing connection to some bricks (likely), (b) the I/O failing on the bricks themselves (unlikely). If most of the i/o is also going to the 3rd brick (since you say the files are already present on all bricks and I/O is successful) , then it is likely to be (a).
How to check this on a gluster infrastructure?
In the fuse mount logs for the engine volume, check if there are any messages for brick disconnects. Something along the lines of "disconnected from volname-client-x". Just guessing here, but maybe even the 'data' volume did experience disconnects and self-heals later but you did not observe it when you ran heal info. See the glustershd log or mount log for for self-heal completion messages on /0-data-replicate-0 /also. Regards, Ravi
Thank you
--------------84EB6DEB84AAFAA57D0538D6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 07/21/2017 02:55 PM, yayo (j) wrote:<br> </div> <blockquote type="cite" cite="mid:CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">2017-07-20 14:48 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000000" bgcolor="#FFFFFF"><span class=""> <p><br> </p> </span> But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case.<br> Hope this helps,<br> Ravi <div> <div class="h5"><br> <br> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5b<wbr>d74327. sources=[0] 1 sinks=2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974<wbr>bcef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974<wbr>bcef81. sources=[0] 1 sinks=2</i></div> </div> </div> </div> </blockquote> </div> </blockquote> </div> </div> </div> </blockquote> </div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> <div class="gmail_signature" data-smartmail="gmail_signature">Hi,</div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> <div class="gmail_signature" data-smartmail="gmail_signature">But we ha1e 2 gluster volume on the same network and the other one (the "Data" gluster) don't have any problems. Why you think there is a network problem?</div> </div> </div> </blockquote> <br> Because pending self-heals come into the picture when I/O from the clients (mounts) do not succeed on some bricks. They are mostly due to <br> (a) the client losing connection to some bricks (likely),<br> (b) the I/O failing on the bricks themselves (unlikely).<br> <br> If most of the i/o is also going to the 3rd brick (since you say the files are already present on all bricks and I/O is successful) , then it is likely to be (a).<br> <br> <blockquote type="cite" cite="mid:CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_signature" data-smartmail="gmail_signature"> How to check this on a gluster infrastructure?</div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> </div> </div> </blockquote> In the fuse mount logs for the engine volume, check if there are any messages for brick disconnects. Something along the lines of "disconnected from volname-client-x".<br> Just guessing here, but maybe even the 'data' volume did experience disconnects and self-heals later but you did not observe it when you ran heal info. See the glustershd log or mount log for for self-heal completion messages on <i> 0-data-replicate-0 </i>also.<br> <br> Regards,<br> Ravi<br> <blockquote type="cite" cite="mid:CAGK=3kx=XufQ3_bbmF1+R3g+gqCa-nTUf05KNbA00eBcHM92+g@mail.gmail.com"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_signature" data-smartmail="gmail_signature">Thank you</div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> <div class="gmail_signature" data-smartmail="gmail_signature"><br> </div> </div> </div> </blockquote> <br> </body> </html> --------------84EB6DEB84AAFAA57D0538D6--
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi, following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it: *gluster peer status on NODE01:* *Number of Peers: 2* *Hostname: dnode02.localdomain.local* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.52* *dnode02.localdomain.local* *10.10.20.90* *10.10.10.20* *gluster peer status on NODE02:* *Number of Peers: 2* *Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10* *Hostname: gdnode04* *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.54* *10.10.10.40* *gluster peer status on NODE04:* *Number of Peers: 2* *Hostname: dnode02.neridom.dom* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *10.10.20.90* *gdnode02* *192.168.10.52* *10.10.10.20* *Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10* All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries? And, what about the selinux? Thank you
Hi, Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots: https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=shar... But the "gluster volume info" command report that all 2 volume are full replicated: *Volume Name: data* *Type: Replicate* *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/data/brick* *Brick2: gdnode02:/gluster/data/brick* *Brick3: gdnode04:/gluster/data/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: enable* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** *server.allow-insecure: on* *Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/engine/brick* *Brick2: gdnode02:/gluster/engine/brick* *Brick3: gdnode04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** server.allow-insecure: on 2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz@gmail.com>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi,
following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:
*gluster peer status on NODE01:* *Number of Peers: 2*
*Hostname: dnode02.localdomain.local* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.52* *dnode02.localdomain.local* *10.10.20.90* *10.10.10.20*
*gluster peer status on NODE02:* *Number of Peers: 2*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
*Hostname: gdnode04* *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.54* *10.10.10.40*
*gluster peer status on NODE04:* *Number of Peers: 2*
*Hostname: dnode02.neridom.dom* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *10.10.20.90* *gdnode02* *192.168.10.52* *10.10.10.20*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
And, what about the selinux?
Thank you
-- Linux User: 369739 http://counter.li.org
This is a multi-part message in MIME format. --------------6CE85A48A5119C9A9913FFCB Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to tell you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=shar...
But the "gluster volume info" command report that all 2 volume are full replicated:
/Volume Name: data/ /Type: Replicate/ /Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d/ /Status: Started/ /Snapshot Count: 0/ /Number of Bricks: 1 x 3 = 3/ /Transport-type: tcp/ /Bricks:/ /Brick1: gdnode01:/gluster/data/brick/ /Brick2: gdnode02:/gluster/data/brick/ /Brick3: gdnode04:/gluster/data/brick/ /Options Reconfigured:/ /nfs.disable: on/ /performance.readdir-ahead: on/ /transport.address-family: inet/ /storage.owner-uid: 36/ /performance.quick-read: off/ /performance.read-ahead: off/ /performance.io-cache: off/ /performance.stat-prefetch: off/ /performance.low-prio-threads: 32/ /network.remote-dio: enable/ /cluster.eager-lock: enable/ /cluster.quorum-type: auto/ /cluster.server-quorum-type: server/ /cluster.data-self-heal-algorithm: full/ /cluster.locking-scheme: granular/ /cluster.shd-max-threads: 8/ /cluster.shd-wait-qlength: 10000/ /features.shard: on/ /user.cifs: off/ /storage.owner-gid: 36/ /features.shard-block-size: 512MB/ /network.ping-timeout: 30/ /performance.strict-o-direct: on/ /cluster.granular-entry-heal: on/ /auth.allow: */ /server.allow-insecure: on/
/Volume Name: engine/ /Type: Replicate/ /Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515/ /Status: Started/ /Snapshot Count: 0/ /Number of Bricks: 1 x 3 = 3/ /Transport-type: tcp/ /Bricks:/ /Brick1: gdnode01:/gluster/engine/brick/ /Brick2: gdnode02:/gluster/engine/brick/ /Brick3: gdnode04:/gluster/engine/brick/ /Options Reconfigured:/ /nfs.disable: on/ /performance.readdir-ahead: on/ /transport.address-family: inet/ /storage.owner-uid: 36/ /performance.quick-read: off/ /performance.read-ahead: off/ /performance.io-cache: off/ /performance.stat-prefetch: off/ /performance.low-prio-threads: 32/ /network.remote-dio: off/ /cluster.eager-lock: enable/ /cluster.quorum-type: auto/ /cluster.server-quorum-type: server/ /cluster.data-self-heal-algorithm: full/ /cluster.locking-scheme: granular/ /cluster.shd-max-threads: 8/ /cluster.shd-wait-qlength: 10000/ /features.shard: on/ /user.cifs: off/ /storage.owner-gid: 36/ /features.shard-block-size: 512MB/ /network.ping-timeout: 30/ /performance.strict-o-direct: on/ /cluster.granular-entry-heal: on/ /auth.allow: */
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz@gmail.com <mailto:jaganz@gmail.com>>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com <mailto:ravishankar@redhat.com>>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
/[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2/ /[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81/ /[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2/
Hi,
following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:
/*gluster peer status on NODE01:*/ /Number of Peers: 2/ / / /Hostname: dnode02.localdomain.local/ /Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd/ /State: Peer in Cluster (Connected)/ /Other names:/ /192.168.10.52/ /dnode02.localdomain.local/ /10.10.20.90/ /10.10.10.20/ / / / / / / / / */gluster peer status on //NODE02:/* /Number of Peers: 2/ / / /Hostname: dnode01.localdomain.local/ /Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12/ /State: Peer in Cluster (Connected)/ /Other names:/ /gdnode01/ /10.10.10.10/ / / /Hostname: gdnode04/ /Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828/ /State: Peer in Cluster (Connected)/ /Other names:/ /192.168.10.54/ /10.10.10.40/ / / /* */ */gluster peer status on //NODE04:/* /Number of Peers: 2/ / / /Hostname: dnode02.neridom.dom/ /Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd/ /State: Peer in Cluster (Connected)/ /Other names:/ /10.10.20.90/ /gdnode02/ /192.168.10.52/ /10.10.10.20/ / / /Hostname: dnode01.localdomain.local/ /Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12/ /State: Peer in Cluster (Connected)/ /Other names:/ /gdnode01/ /10.10.10.10/
/ / / / All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount logs first. -Ravi
Thank you
-- Linux User: 369739 http://counter.li.org
--------------6CE85A48A5119C9A9913FFCB Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 07/21/2017 11:41 PM, yayo (j) wrote:<br> </div> <blockquote type="cite" cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com"> <div dir="ltr">Hi, <div><br> </div> <div>Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots:</div> </div> </blockquote> <br> This is probably some refresh bug in the UI, Sahina might be able to tell you.<br> <blockquote type="cite" cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com"> <div dir="ltr"> <div><br> </div> <div><a href="https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=shar..." moz-do-not-send="true">https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ?usp=sharing</a><br> </div> <div><br> </div> <div>But the "gluster volume info" command report that all 2 volume are full replicated:</div> <div><br> </div> <div><br> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div> <div><i>Volume Name: data</i></div> </div> <div> <div><i>Type: Replicate</i></div> </div> <div> <div><i>Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d</i></div> </div> <div> <div><i>Status: Started</i></div> </div> <div> <div><i>Snapshot Count: 0</i></div> </div> <div> <div><i>Number of Bricks: 1 x 3 = 3</i></div> </div> <div> <div><i>Transport-type: tcp</i></div> </div> <div> <div><i>Bricks:</i></div> </div> <div> <div><i>Brick1: gdnode01:/gluster/data/brick</i></div> </div> <div> <div><i>Brick2: gdnode02:/gluster/data/brick</i></div> </div> <div> <div><i>Brick3: gdnode04:/gluster/data/brick</i></div> </div> <div> <div><i>Options Reconfigured:</i></div> </div> <div> <div><i>nfs.disable: on</i></div> </div> <div> <div><i>performance.readdir-ahead: on</i></div> </div> <div> <div><i>transport.address-family: inet</i></div> </div> <div> <div><i>storage.owner-uid: 36</i></div> </div> <div> <div><i>performance.quick-read: off</i></div> </div> <div> <div><i>performance.read-ahead: off</i></div> </div> <div> <div><i>performance.io-cache: off</i></div> </div> <div> <div><i>performance.stat-prefetch: off</i></div> </div> <div> <div><i>performance.low-prio-threads: 32</i></div> </div> <div> <div><i>network.remote-dio: enable</i></div> </div> <div> <div><i>cluster.eager-lock: enable</i></div> </div> <div> <div><i>cluster.quorum-type: auto</i></div> </div> <div> <div><i>cluster.server-quorum-type: server</i></div> </div> <div> <div><i>cluster.data-self-heal-algorithm: full</i></div> </div> <div> <div><i>cluster.locking-scheme: granular</i></div> </div> <div> <div><i>cluster.shd-max-threads: 8</i></div> </div> <div> <div><i>cluster.shd-wait-qlength: 10000</i></div> </div> <div> <div><i>features.shard: on</i></div> </div> <div> <div><i>user.cifs: off</i></div> </div> <div> <div><i>storage.owner-gid: 36</i></div> </div> <div> <div><i>features.shard-block-size: 512MB</i></div> </div> <div> <div><i>network.ping-timeout: 30</i></div> </div> <div> <div><i>performance.strict-o-direct: on</i></div> </div> <div> <div><i>cluster.granular-entry-heal: on</i></div> </div> <div> <div><i>auth.allow: *</i></div> </div> <div> <div><i>server.allow-insecure: on</i></div> </div> </blockquote> <div><br> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div> <blockquote style="color:rgb(0,0,0);font-size:12.8px;margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"><i>Volume Name: engine</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Type: Replicate</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Volume ID: d19c19e3-910d-437b-8ba7-<wbr>4f2a23d17515</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Status: Started</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Snapshot Count: 0</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Number of Bricks: 1 x 3 = 3</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Transport-type: tcp</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Bricks:</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Brick1: gdnode01:/gluster/engine/brick</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Brick2: gdnode02:/gluster/engine/brick</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Brick3: gdnode04:/gluster/engine/brick</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>Options Reconfigured:</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>nfs.disable: on</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.readdir-ahead: on</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>transport.address-family: inet</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>storage.owner-uid: 36</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.quick-read: off</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.read-ahead: off</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.io-cache: off</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.stat-prefetch: off</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.low-prio-threads: 32</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>network.remote-dio: off</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.eager-lock: enable</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.quorum-type: auto</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.server-quorum-type: server</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.data-self-heal-<wbr>algorithm: full</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.locking-scheme: granular</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.shd-max-threads: 8</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.shd-wait-qlength: 10000</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>features.shard: on</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>user.cifs: off</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>storage.owner-gid: 36</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>features.shard-block-size: 512MB</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>network.ping-timeout: 30</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>performance.strict-o-direct: on</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>cluster.granular-entry-heal: on</i></div> </div> <div class="gmail_extra"> <div class="gmail_quote"><i>auth.allow: *</i></div> </div> </blockquote> <div class="gmail_extra" style="color:rgb(0,0,0);font-size:12.8px"> <div class="gmail_quote"> server.allow-insecure: on</div> </div> </div> <div><br> </div> </div> <div class="gmail_extra"><br> <div class="gmail_quote">2017-07-21 19:13 GMT+02:00 yayo (j) <span dir="ltr"><<a href="mailto:jaganz@gmail.com" target="_blank" moz-do-not-send="true">jaganz@gmail.com</a>></span>:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"><span class="">2017-07-20 14:48 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank" moz-do-not-send="true">ravishankar@redhat.com</a>></span>:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div bgcolor="#FFFFFF"><span class="m_7510525959629396992gmail-"> <p><br> </p> </span> But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case.<br> Hope this helps,<br> Ravi <div> <div class="m_7510525959629396992gmail-h5"><br> <br> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> </div> </div> <blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5b<wbr>d74327. sources=[0] 1 sinks=2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:_<wbr>_afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974<wbr>bcef81</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:a<wbr>fr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974<wbr>bcef81. sources=[0] 1 sinks=2</i></div> </div> </div> </div> </blockquote> </div> </blockquote> </div> </div> </div> </blockquote> <div><br> </div> <div><br> </div> </span> <div>Hi,</div> <div><br> </div> <div>following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:</div> <div><br> </div> </div> </div> <blockquote style="margin:0 0 0 40px;border:none;padding:0px"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><b>gluster peer status on NODE01:</b></i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Peers: 2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Hostname: dnode02.localdomain.local</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Uuid: 7c0ebfa3-5676-4d3f-9bfa-<wbr>7fff6afea0dd</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>State: Peer in Cluster (Connected)</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Other names:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>192.168.10.52</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>dnode02.localdomain.local</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.20.90</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.10.20</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><b><i>gluster peer status on </i><i>NODE02:</i></b></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Peers: 2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Hostname: dnode01.localdomain.local</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Uuid: a568bd60-b3e4-4432-a9bc-<wbr>996c52eaaa12</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>State: Peer in Cluster (Connected)</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Other names:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>gdnode01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.10.10</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Hostname: gdnode04</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Uuid: ce6e0f6b-12cf-4e40-8f01-<wbr>d1609dfc5828</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>State: Peer in Cluster (Connected)</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Other names:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>192.168.10.54</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.10.40</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><b><br> </b></i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><b><i>gluster peer status on </i><i>NODE04:</i></b></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Number of Peers: 2</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Hostname: dnode02.neridom.dom</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Uuid: 7c0ebfa3-5676-4d3f-9bfa-<wbr>7fff6afea0dd</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>State: Peer in Cluster (Connected)</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Other names:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.20.90</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>gdnode02</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>192.168.10.52</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.10.20</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Hostname: dnode01.localdomain.local</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Uuid: a568bd60-b3e4-4432-a9bc-<wbr>996c52eaaa12</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>State: Peer in Cluster (Connected)</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>Other names:</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>gdnode01</i></div> </div> </div> </div> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i>10.10.10.10</i></div> </div> </div> </div> </blockquote> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><i><br> </i></div> <div><i><br> </i></div> <div>All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries? <br> </div> </div> </div> </div> </div> </blockquote> </div> </div> </blockquote> I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?<br> <blockquote type="cite" cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <div><br> </div> </div> <div>And, what about the selinux? <br> </div> </div> </div> </div> </blockquote> </div> </div> </blockquote> Not sure about this. See if there are disconnect messages in the mount logs first.<br> -Ravi<br> <blockquote type="cite" cite="mid:CAGK=3kxB6xGTMxbGanGySXZ5RNRkWOx6GR+iFb867jhF9jHo9Q@mail.gmail.com"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div><br> </div> <div>Thank you</div> <div><br> </div> <div><br> </div> </div> </div> </div> </blockquote> </div> <br> <br clear="all"> <div><br> </div> -- <br> <div class="gmail_signature" data-smartmail="gmail_signature">Linux User: 369739 <a href="http://counter.li.org" target="_blank" moz-do-not-send="true">http://counter.li.org</a></div> </div> </blockquote> <br> </body> </html> --------------6CE85A48A5119C9A9913FFCB--
Hi, Regarding the UI showing incorrect information about engine and data volumes, can you please refresh the UI and see if the issue persists plus any errors in the engine.log files ? Thanks kasturi On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar@redhat.com> wrote:
On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to tell you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFfVmR5aDQ? usp=sharing
But the "gluster volume info" command report that all 2 volume are full replicated:
*Volume Name: data* *Type: Replicate* *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/data/brick* *Brick2: gdnode02:/gluster/data/brick* *Brick3: gdnode04:/gluster/data/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: enable* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** *server.allow-insecure: on*
*Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/engine/brick* *Brick2: gdnode02:/gluster/engine/brick* *Brick3: gdnode04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: **
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz@gmail.com>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi,
following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:
*gluster peer status on NODE01:* *Number of Peers: 2*
*Hostname: dnode02.localdomain.local* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.52* *dnode02.localdomain.local* *10.10.20.90* *10.10.10.20*
*gluster peer status on NODE02:* *Number of Peers: 2*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
*Hostname: gdnode04* *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.54* *10.10.10.40*
*gluster peer status on NODE04:* *Number of Peers: 2*
*Hostname: dnode02.neridom.dom* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *10.10.20.90* *gdnode02* *192.168.10.52* *10.10.10.20*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount logs first. -Ravi
Thank you
-- Linux User: 369739 http://counter.li.org
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi, UI refreshed but problem still remain ... No specific error, I've only these errors but I've read that there is no problem if I have this kind of errors: 2017-07-24 15:53:59,823+02 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] START, GlusterServersListVDSCommand(HostName = node01.localdomain.local, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 29a62417 2017-07-24 15:54:01,066+02 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterServersListVDSCommand, return: [10.10.20.80/24:CONNECTED, node02.localdomain.local:CONNECTED, gdnode04:CONNECTED], log id: 29a62417 2017-07-24 15:54:01,076+02 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] START, GlusterVolumesListVDSCommand(HostName = node01.localdomain.local, GlusterVolumesListVDSParameters:{runAsync='true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 7fce25d3 2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7- 4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,212+02 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode02:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7- 4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,215+02 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode04:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7- 4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,218+02 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e- c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,221+02 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode02:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e- c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,224+02 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode04:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e- c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,224+02 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterVolumesListVDSCommand, return: {d19c19e3-910d-437b-8ba7-4f2a23d17515=org.ovirt.engine.core. common.businessentities.gluster.GlusterVolumeEntity@fdc91062, c7a5dfc9-3e72 -4ea1-843e-c8275d4a7c2d=org.ovirt.engine.core.common.businessentities. gluster.GlusterVolumeEntity@999a6f23}, log id: 7fce25d3 Thank you 2017-07-24 8:12 GMT+02:00 Kasturi Narra <knarra@redhat.com>:
Hi,
Regarding the UI showing incorrect information about engine and data volumes, can you please refresh the UI and see if the issue persists plus any errors in the engine.log files ?
Thanks kasturi
On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar@redhat.com> wrote:
On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to tell you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFf VmR5aDQ?usp=sharing
But the "gluster volume info" command report that all 2 volume are full replicated:
*Volume Name: data* *Type: Replicate* *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/data/brick* *Brick2: gdnode02:/gluster/data/brick* *Brick3: gdnode04:/gluster/data/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: enable* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** *server.allow-insecure: on*
*Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/engine/brick* *Brick2: gdnode02:/gluster/engine/brick* *Brick3: gdnode04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: **
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz@gmail.com>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi,
following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:
*gluster peer status on NODE01:* *Number of Peers: 2*
*Hostname: dnode02.localdomain.local* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.52* *dnode02.localdomain.local* *10.10.20.90* *10.10.10.20*
*gluster peer status on NODE02:* *Number of Peers: 2*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
*Hostname: gdnode04* *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.54* *10.10.10.40*
*gluster peer status on NODE04:* *Number of Peers: 2*
*Hostname: dnode02.neridom.dom* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *10.10.20.90* *gdnode02* *192.168.10.52* *10.10.10.20*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount logs first. -Ravi
Thank you
-- Linux User: 369739 http://counter.li.org
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Linux User: 369739 http://counter.li.org
These errors are because not having glusternw assigned to the correct interface. Once you attach that these errors should go away. This has nothing to do with the problem you are seeing. sahina any idea about engine not showing the correct volume info ? On Mon, Jul 24, 2017 at 7:30 PM, yayo (j) <jaganz@gmail.com> wrote:
Hi,
UI refreshed but problem still remain ...
No specific error, I've only these errors but I've read that there is no problem if I have this kind of errors:
2017-07-24 15:53:59,823+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] START, GlusterServersListVDSCommand(HostName = node01.localdomain.local, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 29a62417 2017-07-24 15:54:01,066+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterServersListVDSCommand, return: [10.10.20.80/24:CONNECTED, node02.localdomain.local:CONNECTED, gdnode04:CONNECTED], log id: 29a62417 2017-07-24 15:54:01,076+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] START, GlusterVolumesListVDSCommand(HostName = node01.localdomain.local, GlusterVolumesListVDSParameters:{runAsync='true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 7fce25d3 2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,212+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode02:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,215+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode04:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,218+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,221+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode02:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,224+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode04:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,224+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterVolumesListVDSCommand, return: {d19c19e3-910d-437 b-8ba7-4f2a23d17515=org.ovirt.engine.core.common.businessentities.gluste r.GlusterVolumeEntity@fdc91062, c7a5dfc9-3e72-4ea1-843e-c8275d 4a7c2d=org.ovirt.engine.core.common.businessentities.gluste r.GlusterVolumeEntity@999a6f23}, log id: 7fce25d3
Thank you
2017-07-24 8:12 GMT+02:00 Kasturi Narra <knarra@redhat.com>:
Hi,
Regarding the UI showing incorrect information about engine and data volumes, can you please refresh the UI and see if the issue persists plus any errors in the engine.log files ?
Thanks kasturi
On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar@redhat.com> wrote:
On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to tell you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFf VmR5aDQ?usp=sharing
But the "gluster volume info" command report that all 2 volume are full replicated:
*Volume Name: data* *Type: Replicate* *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/data/brick* *Brick2: gdnode02:/gluster/data/brick* *Brick3: gdnode04:/gluster/data/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: enable* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** *server.allow-insecure: on*
*Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/engine/brick* *Brick2: gdnode02:/gluster/engine/brick* *Brick3: gdnode04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: **
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz@gmail.com>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi,
following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:
*gluster peer status on NODE01:* *Number of Peers: 2*
*Hostname: dnode02.localdomain.local* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.52* *dnode02.localdomain.local* *10.10.20.90* *10.10.10.20*
*gluster peer status on NODE02:* *Number of Peers: 2*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
*Hostname: gdnode04* *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.54* *10.10.10.40*
*gluster peer status on NODE04:* *Number of Peers: 2*
*Hostname: dnode02.neridom.dom* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *10.10.20.90* *gdnode02* *192.168.10.52* *10.10.10.20*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount logs first. -Ravi
Thank you
-- Linux User: 369739 http://counter.li.org
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Linux User: 369739 http://counter.li.org
On Tue, Jul 25, 2017 at 11:12 AM, Kasturi Narra <knarra@redhat.com> wrote:
These errors are because not having glusternw assigned to the correct interface. Once you attach that these errors should go away. This has nothing to do with the problem you are seeing.
sahina any idea about engine not showing the correct volume info ?
Please provide the vdsm.log (contianing the gluster volume info) and engine.log
On Mon, Jul 24, 2017 at 7:30 PM, yayo (j) <jaganz@gmail.com> wrote:
Hi,
UI refreshed but problem still remain ...
No specific error, I've only these errors but I've read that there is no problem if I have this kind of errors:
2017-07-24 15:53:59,823+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] START, GlusterServersListVDSCommand(HostName = node01.localdomain.local, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 29a62417 2017-07-24 15:54:01,066+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterServersListVDSCommand, return: [10.10.20.80/24:CONNECTED, node02.localdomain.local:CONNECTED, gdnode04:CONNECTED], log id: 29a62417 2017-07-24 15:54:01,076+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] START, GlusterVolumesListVDSCommand(HostName = node01.localdomain.local, GlusterVolumesListVDSParameters:{runAsync= 'true', hostId='4c89baa5-e8f7-4132-a4b3-af332247570c'}), log id: 7fce25d3 2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,212+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode02:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,215+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode04:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,218+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,221+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode02:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,224+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode04:/gluster/data/brick' of volume 'c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' 2017-07-24 15:54:02,224+02 INFO [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [b7590c4] FINISH, GlusterVolumesListVDSCommand, return: {d19c19e3-910d -437b-8ba7-4f2a23d17515=org.ovirt.engine.core. common.businessentities.gluster.GlusterVolumeEntity@fdc91062, c7a5dfc9 -3e72-4ea1-843e-c8275d4a7c2d=org.ovirt.engine.core.c ommon.businessentities.gluster.GlusterVolumeEntity@999a6f23}, log id: 7 fce25d3
Thank you
2017-07-24 8:12 GMT+02:00 Kasturi Narra <knarra@redhat.com>:
Hi,
Regarding the UI showing incorrect information about engine and data volumes, can you please refresh the UI and see if the issue persists plus any errors in the engine.log files ?
Thanks kasturi
On Sat, Jul 22, 2017 at 11:43 AM, Ravishankar N <ravishankar@redhat.com> wrote:
On 07/21/2017 11:41 PM, yayo (j) wrote:
Hi,
Sorry for follow up again, but, checking the ovirt interface I've found that ovirt report the "engine" volume as an "arbiter" configuration and the "data" volume as full replicated volume. Check these screenshots:
This is probably some refresh bug in the UI, Sahina might be able to tell you.
https://drive.google.com/drive/folders/0ByUV7xQtP1gCTE8tUTFf VmR5aDQ?usp=sharing
But the "gluster volume info" command report that all 2 volume are full replicated:
*Volume Name: data* *Type: Replicate* *Volume ID: c7a5dfc9-3e72-4ea1-843e-c8275d4a7c2d* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/data/brick* *Brick2: gdnode02:/gluster/data/brick* *Brick3: gdnode04:/gluster/data/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: enable* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: ** *server.allow-insecure: on*
*Volume Name: engine* *Type: Replicate* *Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515* *Status: Started* *Snapshot Count: 0* *Number of Bricks: 1 x 3 = 3* *Transport-type: tcp* *Bricks:* *Brick1: gdnode01:/gluster/engine/brick* *Brick2: gdnode02:/gluster/engine/brick* *Brick3: gdnode04:/gluster/engine/brick* *Options Reconfigured:* *nfs.disable: on* *performance.readdir-ahead: on* *transport.address-family: inet* *storage.owner-uid: 36* *performance.quick-read: off* *performance.read-ahead: off* *performance.io-cache: off* *performance.stat-prefetch: off* *performance.low-prio-threads: 32* *network.remote-dio: off* *cluster.eager-lock: enable* *cluster.quorum-type: auto* *cluster.server-quorum-type: server* *cluster.data-self-heal-algorithm: full* *cluster.locking-scheme: granular* *cluster.shd-max-threads: 8* *cluster.shd-wait-qlength: 10000* *features.shard: on* *user.cifs: off* *storage.owner-gid: 36* *features.shard-block-size: 512MB* *network.ping-timeout: 30* *performance.strict-o-direct: on* *cluster.granular-entry-heal: on* *auth.allow: **
server.allow-insecure: on
2017-07-21 19:13 GMT+02:00 yayo (j) <jaganz@gmail.com>:
2017-07-20 14:48 GMT+02:00 Ravishankar N <ravishankar@redhat.com>:
But it does say something. All these gfids of completed heals in the log below are the for the ones that you have given the getfattr output of. So what is likely happening is there is an intermittent connection problem between your mount and the brick process, leading to pending heals again after the heal gets completed, which is why the numbers are varying each time. You would need to check why that is the case. Hope this helps, Ravi
*[2017-07-20 09:58:46.573079] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2* *[2017-07-20 09:59:22.995003] I [MSGID: 108026] [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81* *[2017-07-20 09:59:22.999372] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=[0] 1 sinks=2*
Hi,
following your suggestion, I've checked the "peer" status and I found that there is too many name for the hosts, I don't know if this can be the problem or part of it:
*gluster peer status on NODE01:* *Number of Peers: 2*
*Hostname: dnode02.localdomain.local* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.52* *dnode02.localdomain.local* *10.10.20.90* *10.10.10.20*
*gluster peer status on NODE02:* *Number of Peers: 2*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
*Hostname: gdnode04* *Uuid: ce6e0f6b-12cf-4e40-8f01-d1609dfc5828* *State: Peer in Cluster (Connected)* *Other names:* *192.168.10.54* *10.10.10.40*
*gluster peer status on NODE04:* *Number of Peers: 2*
*Hostname: dnode02.neridom.dom* *Uuid: 7c0ebfa3-5676-4d3f-9bfa-7fff6afea0dd* *State: Peer in Cluster (Connected)* *Other names:* *10.10.20.90* *gdnode02* *192.168.10.52* *10.10.10.20*
*Hostname: dnode01.localdomain.local* *Uuid: a568bd60-b3e4-4432-a9bc-996c52eaaa12* *State: Peer in Cluster (Connected)* *Other names:* *gdnode01* *10.10.10.10*
All these ip are pingable and hosts resolvible across all 3 nodes but, only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount logs first. -Ravi
Thank you
-- Linux User: 369739 http://counter.li.org
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Linux User: 369739 http://counter.li.org
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
2017-07-25 7:42 GMT+02:00 Kasturi Narra <knarra@redhat.com>:
These errors are because not having glusternw assigned to the correct interface. Once you attach that these errors should go away. This has nothing to do with the problem you are seeing.
Hi, You talking about errors like these? 2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a' How to assign "glusternw (???)" to the correct interface? Other errors on unsync gluster elements still remain... This is a production env, so, there is any chance to subscribe to RH support? Thank you
On Tue, Jul 25, 2017 at 1:45 PM, yayo (j) <jaganz@gmail.com> wrote:
2017-07-25 7:42 GMT+02:00 Kasturi Narra <knarra@redhat.com>:
These errors are because not having glusternw assigned to the correct interface. Once you attach that these errors should go away. This has nothing to do with the problem you are seeing.
Hi,
You talking about errors like these?
2017-07-24 15:54:02,209+02 WARN [org.ovirt.engine.core.vdsbro ker.gluster.GlusterVolumesListReturn] (DefaultQuartzScheduler2) [b7590c4] Could not associate brick 'gdnode01:/gluster/engine/brick' of volume 'd19c19e3-910d-437b-8ba7-4f2a23d17515' with correct network as no gluster network found in cluster '00000002-0002-0002-0002-00000000017a'
How to assign "glusternw (???)" to the correct interface?
https://www.ovirt.org/blog/2017/04/up-and-running-with-ovirt-4.1-and-gluster... "Storage network" section explains this. Please make sure that gdnode01 is resolvable from engine.
Other errors on unsync gluster elements still remain... This is a production env, so, there is any chance to subscribe to RH support?
The unsynced entries - did you check for disconnect messages in the mount log as suggested by Ravi? For Red Hat support, the best option is to contact your local Red Hat representative.
Thank you
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
2017-07-25 11:31 GMT+02:00 Sahina Bose <sabose@redhat.com>:
Other errors on unsync gluster elements still remain... This is a production env, so, there is any chance to subscribe to RH support?
The unsynced entries - did you check for disconnect messages in the mount log as suggested by Ravi?
Hi have provided this (check past mails): * tail -f /var/log/glusterfs/rhev-data-center-mnt-glusterSD-dvirtgluster\:engine.log* Is enougth? Thank you
All these ip are pingable and hosts resolvible across all 3 nodes but,
only the 10.10.10.0 network is the decidated network for gluster (rosolved using gdnode* host names) ... You think that remove other entries can fix the problem? So, sorry, but, how can I remove other entries?
I don't think having extra entries could be a problem. Did you check the fuse mount logs for disconnect messages that I referred to in the other email?
* tail -f /var/log/glusterfs/rhev-data-center-mnt-glusterSD-dvirtgluster\:engine.log* *NODE01:* [2017-07-24 07:34:00.799347] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0 -glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport endpoint is not connected) [2017-07-24 07:44:46.687334] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0 -glusterfsd-mgmt: Exhausted all volfile servers [2017-07-24 09:04:25.951350] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0 -glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport endpoint is not connected) [2017-07-24 09:15:11.839357] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0 -glusterfsd-mgmt: Exhausted all volfile servers [2017-07-24 10:34:51.231353] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0 -glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport endpoint is not connected) [2017-07-24 10:45:36.991321] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0 -glusterfsd-mgmt: Exhausted all volfile servers [2017-07-24 12:05:16.383323] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0 -glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport endpoint is not connected) [2017-07-24 12:16:02.271320] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0 -glusterfsd-mgmt: Exhausted all volfile servers [2017-07-24 13:35:41.535308] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0 -glusterfsd-mgmt: failed to connect with remote-host: gdnode03 (Transport endpoint is not connected) [2017-07-24 13:46:27.423304] I [glusterfsd-mgmt.c:1926:mgmt_rpc_notify] 0 -glusterfsd-mgmt: Exhausted all volfile servers Why again gdnode03? Was removed from gluster! was the arbiter node... *NODE02:* [2017-07-24 14:08:18.709209] I [MSGID: 108026] [ afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00-fd5b-4326-a879-326ff56181de. sources=0 [ 1] sinks=2 [2017-07-24 14:08:38.746688] I [MSGID: 108026] [ afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81 [2017-07-24 14:08:38.749379] I [MSGID: 108026] [ afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=0 [1] sinks=2 [2017-07-24 14:08:46.068001] I [MSGID: 108026] [ afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00-fd5b-4326-a879-326ff56181de. sources=0 [ 1] sinks=2 The message "I [MSGID: 108026] [ afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81" repeated 3 times between [2017-07-24 14:08:38.746688] and [2017-07-24 14:10:09.088625] The message "I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=0 [1] sinks=2 " repeated 3 times between [2017-07-24 14:08:38.749379] and [2017-07-24 14:10:09.091377] [2017-07-24 14:10:19.384379] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00-fd5b-4326-a879-326ff56181de. sources=0 [1] sinks=2 [2017-07-24 14:10:39.433155] I [MSGID: 108026] [afr-self-heal-metadata.c:51: __afr_selfheal_metadata_do] 0-engine-replicate-0: performing metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81 [2017-07-24 14:10:39.435847] I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed metadata selfheal on f05b9742-2771-484a-85fc-5b6974bcef81. sources=0 [1] sinks=2 *NODE04:* [2017-07-24 14:08:56.789598] I [MSGID: 108026] [afr-self-heal-common.c:1254 :afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2 [2017-07-24 14:09:17.231987] I [MSGID: 108026] [afr-self-heal-common.c:1254 :afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00 -fd5b-4326-a879-326ff56181de. sources=[0] 1 sinks=2 [2017-07-24 14:09:38.039541] I [MSGID: 108026] [afr-self-heal-common.c:1254 :afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2 [2017-07-24 14:09:48.875602] I [MSGID: 108026] [afr-self-heal-common.c:1254 :afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on db56ac00 -fd5b-4326-a879-326ff56181de. sources=[0] 1 sinks=2 [2017-07-24 14:10:39.832068] I [MSGID: 108026] [afr-self-heal-common.c:1254 :afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2 The message "I [MSGID: 108026] [afr-self-heal-common.c:1254:afr_log_selfheal] 0-engine-replicate-0: Completed data selfheal on e6dfd556-340b-4b76-b47b-7b6f5bd74327. sources=[0] 1 sinks=2 " repeated 3 times between [2017-07-24 14:10: 39.832068] and [2017-07-24 14:12:22.686142] Last message was (I think) because I have reexecute an "heal" command n.b. dvirtgluster is the RR DNS for all node gluster
And, what about the selinux?
Not sure about this. See if there are disconnect messages in the mount logs first. -Ravi
Thank you
No messages selinux related... Thank you!
2017-07-19 11:22 GMT+02:00 yayo (j) <jaganz@gmail.com>:
running the "gluster volume heal engine" don't solve the problem...
Some extra info:
We have recently changed the gluster from: 2 (full repliacated) + 1 arbiter to 3 full replicated cluster but i don't know this is the problem...
Hi, I'm sorry for the follow up. I want to say that after upgrade all nodes to the same level, all problems are solved and cluster works very well now! Thank you all for the support!
participants (4)
-
Kasturi Narra -
Ravishankar N -
Sahina Bose -
yayo (j)