Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements

Wednesday, 19 July 2017

This is a multi-part message in MIME format.
--------------0C293E7D7C0723BAFB7F6D00
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

On 07/19/2017 08:02 PM, Sahina Bose wrote:
...
 [Adding gluster-users]

 On Wed, Jul 19, 2017 at 2:52 PM, yayo (j) <jaganz(a)gmail.com 
 <mailto:jaganz@gmail.com>> wrote:

     Hi all,

     We have an ovirt cluster hyperconverged with hosted engine on 3
     full replicated node . This cluster have 2 gluster volume:

     - data: volume for the Data (Master) Domain (For vm)
     - engine: volume fro the hosted_storage  Domain (for hosted engine)

     We have this problem: "engine" gluster volume have always unsynced
     elements and we cant' fix the problem, on command line we have
     tried to use the "heal" command but elements remain always
     unsynced ....

     Below the heal command "status":

     [root@node01 ~]# gluster volume heal engine info
     Brick node01:/gluster/engine/brick
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.48
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.64
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.60
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.2
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.68

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.61
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.1
     /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
     /.shard/8aa74564-6740-403e-ad51-f56d9ca5d7a7.20
     /__DIRECT_IO_TEST__
     Status: Connected
     Number of entries: 12

     Brick node02:/gluster/engine/brick

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/19d71267-52a4-42a3-bb1e-e3145361c0c2/7a215635-02f3-47db-80db-8b689c6a8f01
     <gfid:9a601373-bbaa-44d8-b396-f0b9b12c026f>
     /8f215dd2-8531-4a4f-b6ed-ea789dd8821b/dom_md/ids
     <gfid:1e309376-c62e-424f-9857-f9a0c3a729bf>
     <gfid:e3565b50-1495-4e5b-ae88-3bceca47b7d9>
     <gfid:4e33ac33-dddb-4e29-b4a3-51770b81166a>
     /__DIRECT_IO_TEST__
     <gfid:67606789-1f34-4c15-86b8-c0d05b07f187>
     <gfid:9ef88647-cfe6-4a35-a38c-a5173c9e8fc0>

/8f215dd2-8531-4a4f-b6ed-ea789dd8821b/images/88d41053-a257-4272-9e2e-2f3de0743b81/6573ed08-d3ed-4d12-9227-2c95941e1ad6
     <gfid:9ad720b2-507d-4830-8294-ec8adee6d384>
     <gfid:d9853e5d-a2bf-4cee-8b39-7781a98033cf>
     Status: Connected
     Number of entries: 12

     Brick node04:/gluster/engine/brick
     Status: Connected
     Number of entries: 0

     running the "gluster volume heal engine" don't solve the problem...

1. What does the glustershd.log say on all 3 nodes when you run the 
command? Does it complain anything about these files?
2. Are these 12 files also present in the 3rd data brick?
3. Can you provide the output of `gluster volume info` for the this volume?
...

     Some extra info:

     We have recently changed the gluster from: 2 (full repliacated) +
     1 arbiter to 3 full replicated cluster

Just curious, how did you do this? `remove-brick` of arbiter brick 
followed by an `add-brick` to increase to replica-3?

Thanks,
Ravi
...

     but i don't know this is the problem...

     The "data" volume is good and healty and have no unsynced entry.

     Ovirt refuse to put the node02 and node01 in "maintenance mode"
     and complains about "unsynced elements"

     How can I fix this?
     Thank you

     _______________________________________________
     Users mailing list
     Users(a)ovirt.org <mailto:Users@ovirt.org>
     http://lists.ovirt.org/mailman/listinfo/users
     <http://lists.ovirt.org/mailman/listinfo/users>

--------------0C293E7D7C0723BAFB7F6D00
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 07/19/2017 08:02 PM, Sahina Bose
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com">
      <div dir="ltr">[Adding gluster-users]<br>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, Jul 19, 2017 at 2:52 PM, yayo
          (j) <span dir="ltr">&lt;<a
href="mailto:jaganz@gmail.com"
              target="_blank"
moz-do-not-send=&quot;true&quot;&gt;jaganz(a)gmail.com&lt;/a&gt;&amp;gt;&lt;/span&gt;
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">Hi all,
              <div><br>
              </div>
              <div>We have an ovirt cluster hyperconverged with hosted
                engine on 3 full replicated node . This cluster have 2
                gluster volume:</div>
              <div><br>
              </div>
              <div>- data: volume for the Data (Master) Domain (For vm)</div>
              <div>- engine: volume fro the hosted_storage  Domain (for
                hosted engine)</div>
              <div><br>
              </div>
              <div>We have this problem: "engine" gluster volume have
                always unsynced elements and we cant' fix the problem,
                on command line we have tried to use the "heal" command
                but elements remain always unsynced .... </div>
              <div><br>
              </div>
              <div>Below the heal command "status":</div>
              <div><br>
              </div>
              <div>
                <div class="m_6699510126617823271gmail-hljs
                  m_6699510126617823271gmail-objectivec"
style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace">
                  <div>[root@node01 ~]<span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"># gluster volume
                      heal engine info</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Brick
                      node01:/gluster/engine/brick</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.48</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.64</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.60</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.2</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.68</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/19d71267-<wbr>52a4-42a3-bb1e-e3145361c0c2/<wbr>7a215635-02f3-47db-80db-<wbr>8b689c6a8f01</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/88d41053-<wbr>a257-4272-9e2e-2f3de0743b81/<wbr>6573ed08-d3ed-4d12-9227-<wbr>2c95941e1ad6</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.61</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.1</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/dom_md/ids</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/.shard/8aa74564-6740-403e-<wbr>ad51-f56d9ca5d7a7.20</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/__DIRECT_IO_TEST__</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Status:
Connected</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Number of entries:
                      12</span></div>
                  <div><br>
                  </div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Brick
                      node02:/gluster/engine/brick</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/19d71267-<wbr>52a4-42a3-bb1e-e3145361c0c2/<wbr>7a215635-02f3-47db-80db-<wbr>8b689c6a8f01</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:9a601373-bbaa-44d8-b396-<wbr>f0b9b12c026f&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/dom_md/ids</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:1e309376-c62e-424f-9857-<wbr>f9a0c3a729bf&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:e3565b50-1495-4e5b-ae88-<wbr>3bceca47b7d9&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:4e33ac33-dddb-4e29-b4a3-<wbr>51770b81166a&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/__DIRECT_IO_TEST__</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:67606789-1f34-4c15-86b8-<wbr>c0d05b07f187&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:9ef88647-cfe6-4a35-a38c-<wbr>a5173c9e8fc0&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"

style="color:rgb(31,113,153)">/8f215dd2-8531-4a4f-b6ed-<wbr>ea789dd8821b/images/88d41053-<wbr>a257-4272-9e2e-2f3de0743b81/<wbr>6573ed08-d3ed-4d12-9227-<wbr>2c95941e1ad6</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:9ad720b2-507d-4830-8294-<wbr>ec8adee6d384&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)"><span
                        class="m_6699510126617823271gmail-hljs-meta-string"

style="color:rgb(77,153,191)">&lt;gfid:d9853e5d-a2bf-4cee-8b39-<wbr>7781a98033cf&gt;</span></span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Status:
Connected</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Number of entries:
                      12</span></div>
                  <div><br>
                  </div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Brick
                      node04:/gluster/engine/brick</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Status:
Connected</span></div>
                  <div><span
                      class="m_6699510126617823271gmail-hljs-meta"
                      style="color:rgb(31,113,153)">Number of entries:
0</span></div>
                  <div><br>
                  </div>
                </div>
              </div>
              <br>
              <div>  </div>
              <div>running the "gluster volume heal engine" don't
solve
                the problem...<br>
              </div>
              <div><br>
              </div>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
    1. What does the glustershd.log say on all 3 nodes when you run the
    command? Does it complain anything about these files?<br>
    2. Are these 12 files also present in the 3rd data brick?<br>
    3. Can you provide the output of `gluster volume info` for the this
    volume?<br>
    <blockquote type="cite"
cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com">
      <div class="gmail_extra">
        <div class="gmail_quote">
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">
              <div>Some extra info:</div>
              <div><br>
              </div>
              <div>We have recently changed the gluster from: 2 (full
                repliacated) + 1 arbiter to 3 full replicated cluster </div>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
    Just curious, how did you do this? `remove-brick` of arbiter brick 
    followed by an `add-brick` to increase to replica-3?<br>
    <br>
    Thanks,<br>
    Ravi<br>
    <blockquote type="cite"
cite="mid:CACjzOvcxmzxTxynS5fUs5HyEGN-AYF3P6Y34qLJ6MjktQh9djA@mail.gmail.com">
      <div class="gmail_extra">
        <div class="gmail_quote">
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">
              <div>but i don't know this is the problem...</div>
              <div><br>
              </div>
              <div>The "data" volume is good and healty and have no
                unsynced entry.</div>
              <div><br>
              </div>
              <div>Ovirt refuse to put the node02 and node01 in
                "maintenance mode" and complains about "unsynced
                elements"<br>
              </div>
              <div><br>
              </div>
              <div>How can I fix this?</div>
              <div>Thank you </div>
            </div>
            <br>
            ______________________________<wbr>_________________<br>
            Users mailing list<br>
            <a href="mailto:Users@ovirt.org"
moz-do-not-send=&quot;true&quot;&gt;Users(a)ovirt.org&lt;/a&gt;&lt;br&gt;
            <a href="http://lists.ovirt.org/mailman/listinfo/users"
              rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/<wbr>mai...
            <br>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------0C293E7D7C0723BAFB7F6D00--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] ovirt 4.1 hosted engine hyper converged on glusterfs 3.8.10 : "engine" storage domain alway complain about "unsynced" elements