<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="moz-cite-prefix">Le 19/12/2016 à 08:28, Sahina Bose a
      écrit :<br>
    </div>
    <blockquote
cite="mid:CACjzOvdhyA2-n5MyyVRAgNabzx1NHVRuV-h-hxZSrb3PtYHPVA@mail.gmail.com"
      type="cite">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Fri, Dec 16, 2016 at 11:00 PM,
            Nathanaël Blanchet <span dir="ltr">&lt;<a
                moz-do-not-send="true" href="mailto:blanchet@abes.fr"
                target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>&gt;</span> wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"><span class="">
                  <p><br>
                  </p>
                  <br>
                  <div class="m_-152671267708035808moz-cite-prefix">Le
                    16/12/2016 à 16:34, Sahina Bose a écrit :<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div>
                        <div>Failed to find host
                          'Host[guadalupe1,7a30c899-<wbr>a317-479a-b07b-244bc2374485]'
                          in gluster peer list from
                          'Host[guadalupe1,7a30c899-<wbr>a317-479a-b07b-244bc2374485]'
                          on attempt 2<br>
                          It looks the gluster uuid  saved in the ovirt
                          engine db does not match the one returned from
                          CLI<br>
                          <br>
                        </div>
                        Was this host reinstalled? <br>
                      </div>
                      <div>You may need to remove host from engine and
                        add it again. If that doesn't work you may need
                        to manually change the uuid value in the
                        database (gluster_server table)<br>
                      </div>
                    </div>
                  </blockquote>
                </span> Removing host did nothing, indeed I had to go to
                the gluster_server table to remove any disconnected host
                uuid, but it was not enough. Then I had then to remove
                the host and reinstall it as a new host.<br>
                Thank you, I've been spending a lot of time to solve
                this issue.</div>
            </blockquote>
            <div><br>
            </div>
            <div>Sorry to hear that you had trouble with this. Could you
              explain a bit on how you got into this state?<br>
              <br>
            </div>
            <div>Was it because you re-provisioned one of the gluster
              nodes and the gluster UUID was reset (without oVirt being
              aware of it?). Would like to either fix/enhance the engine
              to handle this if it's a common enough use-case<br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    When going to the gluster_server table, I realized that there were
    some (disconnected) hosts probed with the gluster network IP. A the
    begining, I didn't use the gluster network so my hosts were probed
    on the management network and all was fine. When I decided to change
    the gluster traffic to a dedicated network (you answered to me about
    it there :
    <a class="moz-txt-link-freetext" href="https://www.mail-archive.com/users@ovirt.org/msg37742.html">https://www.mail-archive.com/users@ovirt.org/msg37742.html</a>), I
    believed that hosts would be probed with the new network IP, but
    they didn't. So I manually probed them with the gluster IP, and I
    think all my troubles come from there. I reinstalled vdsm, and then
    nothing was ok since since this moment.<br>
    <blockquote
cite="mid:CACjzOvdhyA2-n5MyyVRAgNabzx1NHVRuV-h-hxZSrb3PtYHPVA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF">
                <div>
                  <div class="h5"><br>
                    <blockquote type="cite">
                      <div class="gmail_extra"><br>
                        <div class="gmail_quote">On Fri, Dec 16, 2016 at
                          7:00 PM, Nathanaël Blanchet <span dir="ltr">&lt;<a
                              moz-do-not-send="true"
                              class="m_-152671267708035808moz-txt-link-abbreviated"
                              href="mailto:blanchet@abes.fr"
                              target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>&gt;</span>
                          wrote:<br>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div text="#000000" bgcolor="#FFFFFF">
                              extract of the last engine logs, thank you
                              <div>
                                <div class="m_-152671267708035808h5"><br>
                                  <br>
                                  <div
                                    class="m_-152671267708035808m_8928738385687730066moz-cite-prefix">Le
                                    16/12/2016 à 14:02, Sahina Bose a
                                    écrit :<br>
                                  </div>
                                  <blockquote type="cite">
                                    <div dir="ltr">Could you attach the
                                      engine log with this error?<br>
                                    </div>
                                    <div class="gmail_extra"><br>
                                      <div class="gmail_quote">On Fri,
                                        Dec 16, 2016 at 4:29 PM,
                                        Nathanaël Blanchet <span
                                          dir="ltr">&lt;<a
                                            moz-do-not-send="true"
                                            class="m_-152671267708035808moz-txt-link-abbreviated"
href="mailto:blanchet@abes.fr" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>&gt;</span>
                                        wrote:<br>
                                        <blockquote class="gmail_quote"
                                          style="margin:0 0 0
                                          .8ex;border-left:1px #ccc
                                          solid;padding-left:1ex">Hi,<br>
                                          <br>
                                          I used to successfully run a
                                          replica 3 gluster volume, but
                                          since the last 4.0.5 update,
                                          they can't connect each other
                                          with the message : gluster
                                          [gluster peer status <a
                                            moz-do-not-send="true"
                                            href="http://guadalupe1.v100.abes.fr"
                                            rel="noreferrer"
                                            target="_blank">guadalupe1.v100.abes.fr</a>]
                                          command failed on server <a
                                            moz-do-not-send="true"
                                            href="http://guadalupe2.v100.abes.fr"
                                            rel="noreferrer"
                                            target="_blank">guadalupe2.v100.abes.fr</a>.<br>
                                          <br>
                                          So host guadalupe1 can't never
                                          be up.<br>
                                          <br>
                                          When doing gluster peer probe,
                                          they are connected as
                                          expected. I reinstalled vdsm
                                          and gluster, but it is still
                                          the same.<br>
                                          <br>
                                          I found this on guadalupe2
                                          supervdsm.log<br>
                                          <br>
                                          MainProcess|jsonrpc.Executor/6<wbr>::DEBUG::2016-12-16
                                          11:53:21,429::supervdsmServer:<wbr>:99::SuperVdsm.ServerCallback:<wbr>:(wrapper)
                                          return peerStatus with
                                          [{'status': 'CONNECTED',
                                          'hostname': '<a
                                            moz-do-not-send="true"
                                            href="http://10.34.101.56/24"
                                            rel="noreferrer"
                                            target="_blank">10.34.101.56/24</a>',
                                          'uuid':
                                          'c259c09b-8d7c-4b12-8745-67719<wbr>9877583'},
                                          {'status': 'CONNECTED',
                                          'hostname': '<a
                                            moz-do-not-send="true"
                                            href="http://guadalupe3.v100.abes.fr"
                                            rel="noreferrer"
                                            target="_blank">guadalupe3.v100.abes.fr</a>',
                                          'uuid':
                                          '6af67cd3-7931-446d-aaa2-ffea5<wbr>1325adc'},
                                          {'status': 'CONNECTED',
                                          'hostname': '<a
                                            moz-do-not-send="true"
                                            href="http://guadalupe1.v100.abes.fr"
                                            rel="noreferrer"
                                            target="_blank">guadalupe1.v100.abes.fr</a>',
                                          'uuid':
                                          '8eb485cd-31c4-4c3a-a315-3dc6d<wbr>3ddc0c9'}]<br>
                                          MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
                                          11:53:21,490::supervdsmServer:<wbr>:92::SuperVdsm.ServerCallback:<wbr>:(wrapper)
                                          call peerProbe with () {}<br>
                                          MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
                                          11:53:21,491::commands::68::ro<wbr>ot::(execCmd)
                                          /usr/bin/taskset --cpu-list
                                          0-63 /usr/sbin/gluster
                                          --mode=script peer probe <a
                                            moz-do-not-send="true"
                                            href="http://guadalupe1.v100.abes.fr"
                                            rel="noreferrer"
                                            target="_blank">guadalupe1.v100.abes.fr</a>
                                          --xml (cwd None)<br>
                                          MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
                                          11:53:21,570::commands::86::ro<wbr>ot::(execCmd)
                                          SUCCESS: &lt;err&gt; = '';
                                          &lt;rc&gt; = 0<br>
                                          MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
                                          11:53:21,570::supervdsmServer:<wbr>:99::SuperVdsm.ServerCallback:<wbr>:(wrapper)
                                          return peerProbe with True<br>
                                          <br>
                                          We can see guadalupe2 can see
                                          guadalupe1 but taskset still
                                          executes peer probe to
                                          guadalupe1 with message "Host
                                          <a moz-do-not-send="true"
                                            href="http://guadalupe1.v100.abes.fr"
                                            rel="noreferrer"
                                            target="_blank">guadalupe1.v100.abes.fr</a>
                                          port 24007 already in peer
                                          list"<br>
                                          <br>
                                          How can I say to guadalupe2
                                          stop trying to probe
                                          guadalupe1?<br>
                                          <br>
                                          <br>
                                          -- <br>
                                          Nathanaël Blanchet<br>
                                          <br>
                                          Supervision réseau<br>
                                          Pôle Infrastrutures
                                          Informatiques<br>
                                          227 avenue
                                          Professeur-Jean-Louis-Viala<br>
                                          34193 MONTPELLIER CEDEX 5     
                                           <br>
                                          Tél. 33 (0)4 67 54 84 55<br>
                                          Fax  33 (0)4 67 54 84 14<br>
                                          <a moz-do-not-send="true"
                                            href="mailto:blanchet@abes.fr"
                                            target="_blank">blanchet@abes.fr</a><br>
                                          <br>
                                          ______________________________<wbr>_________________<br>
                                          Users mailing list<br>
                                          <a moz-do-not-send="true"
                                            href="mailto:Users@ovirt.org"
                                            target="_blank">Users@ovirt.org</a><br>
                                          <a moz-do-not-send="true"
                                            href="http://lists.ovirt.org/mailman/listinfo/users"
                                            rel="noreferrer"
                                            target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
                                        </blockquote>
                                      </div>
                                      <br>
                                    </div>
                                  </blockquote>
                                  <br>
                                  <pre class="m_-152671267708035808m_8928738385687730066moz-signature" cols="72">-- 
Nathanaël Blanchet

Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5         
Tél. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a moz-do-not-send="true" class="m_-152671267708035808m_8928738385687730066moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr" target="_blank">blanchet@abes.fr</a> </pre>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                        <br>
                      </div>
                    </blockquote>
                    <br>
                    <pre class="m_-152671267708035808moz-signature" cols="72">-- 
Nathanaël Blanchet

Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5         
Tél. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a moz-do-not-send="true" class="m_-152671267708035808moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr" target="_blank">blanchet@abes.fr</a> </pre>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">-- 
Nathanaël Blanchet

Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5         
Tél. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a> </pre>
  </body>
</html>