<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <br>
    <div class="moz-cite-prefix">On 07/18/2014 05:43 PM, Andrew Lau
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAD7dF9dRMRa_hMOg0qvKAjm1LkyuunR5sHZwBizJnscfJwssJQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_default" style="font-family:tahoma,sans-serif">&#8203;
          &#8203;</div>
        <div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
        </div>
        <div class="gmail_extra">
          <div class="gmail_quote">On Fri, Jul 18, 2014 at 10:06 PM,
            Vijay Bellur <span dir="ltr">&lt;<a moz-do-not-send="true"
                href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding
              gluster-devel]
              <div class=""><br>
                <br>
                On 07/18/2014 05:20 PM, Andrew Lau wrote:<br>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi
                  all,<br>
                  <br>
                  As most of you have got hints from previous messages,
                  hosted engine<br>
                  won't work on gluster . A quote from BZ1097639<br>
                  <br>
                  "Using hosted engine with Gluster backed storage is
                  currently something<br>
                  we really warn against.<br>
                  <br>
                  <br>
                  I think this bug should be closed or re-targeted at
                  documentation, because there is nothing we can do
                  here. Hosted engine assumes that all writes are atomic
                  and (immediately) available for all hosts in the
                  cluster. Gluster violates those assumptions.<br>
                  &#8203;"<br>
                </blockquote>
              </div>
              I tried going through BZ1097639 but could not find much
              detail with respect to gluster there.<br>
              <br>
              A few questions around the problem:<br>
              <br>
              1. Can somebody please explain in detail the scenario that
              causes the problem?<br>
              <br>
              2. Is hosted engine performing synchronous writes to
              ensure that writes are durable?<br>
              <br>
              Also, if there is any documentation that details the
              hosted engine architecture that would help in enhancing
              our understanding of its interactions with gluster.
              <div class=""><br>
                <br>
                <blockquote class="gmail_quote" style="margin:0px 0px
                  0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">&#8203;<br>
                  <br>
                  Now my question, does this theory prevent a scenario
                  of perhaps<br>
                  something like a gluster replicated volume being
                  mounted as a glusterfs<br>
                  filesystem and then re-exported as the native kernel
                  NFS share for the<br>
                  hosted-engine to consume? It could then be possible to
                  chuck ctdb in<br>
                  there to provide a last resort failover solution. I
                  have tried myself<br>
                  and suggested it to two people who are running a
                  similar setup. Now<br>
                  using the native kernel NFS server for hosted-engine
                  and they haven't<br>
                  reported as many issues. Curious, could anyone
                  validate my theory on this?<br>
                  <br>
                </blockquote>
                <br>
              </div>
              If we obtain more details on the use case and obtain
              gluster logs from the failed scenarios, we should be able
              to understand the problem better. That could be the first
              step in validating your theory or evolving further
              recommendations :).<br>
              <br>
            </blockquote>
            <div><br>
            </div>
            <div>
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif">&#8203;I'm not sure how
                useful this is, but &#8203;Jiri Moskovcak tracked this down in
                an off list message.</div>
              <br>
            </div>
            <div>
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif">&#8203;Message Quote:&#8203;</div>
              <br>
            </div>
            <div>
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif">&#8203;==&#8203;</div>
              <br>
            </div>
            <div>
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif">
                &#8203;<span
                  style="font-family:arial,sans-serif;font-size:13px">We
                  were able to track it down to this (thanks Andrew for
                  providing the testing setup):</span></div>
              <br style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><span
                style="font-family:arial,sans-serif;font-size:13px">h</span><span
                style="font-family:arial,sans-serif;font-size:13px">a_agent
                service_type=</span><span class=""
                style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
                style="font-family:arial,sans-serif;font-size:13px">-</span><span
                class=""
                style="font-family:arial,sans-serif;font-size:13px">engine</span><span
                style="font-family:arial,sans-serif;font-size:13px">'</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">Traceback
                (most recent call last):</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                File "/usr/lib/python2.6/site-</span><span
                style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
                style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",
                line 165, in handle</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                &nbsp; response = "success " + self._dispatch(data)</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                File "/usr/lib/python2.6/site-</span><span
                style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
                style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",
                line 261, in _dispatch</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                &nbsp; .get_all_stats_for_service_</span><span
                style="font-family:arial,sans-serif;font-size:13px">typ</span><span
                style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                File "/usr/lib/python2.6/site-</span><span
                style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
                style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",
                line 41, in get_all_stats_for_service_type</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                &nbsp; d = self.get_raw_stats_for_</span><span
                style="font-family:arial,sans-serif;font-size:13px">service</span><span
                style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir,
                service_type)</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                File "/usr/lib/python2.6/site-</span><span
                style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
                style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",
                line 74, in get_raw_stats_for_service_type</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">&nbsp;
                &nbsp; f = os.open(path, direct_flag | os.O_RDONLY)</span><br
                style="font-family:arial,sans-serif;font-size:13px">
              <span style="font-family:arial,sans-serif;font-size:13px">OSError:
                [Errno 116] Stale file handle: '/rhev/data-center/mnt/</span><span
                style="font-family:arial,sans-serif;font-size:13px">localho</span><span
                style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span
                class=""
                style="font-family:arial,sans-serif;font-size:13px">engine</span><span
                style="font-family:arial,sans-serif;font-size:13px">/</span><span
                style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span
                style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><span
                style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span
                style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span
                class=""
                style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
                style="font-family:arial,sans-serif;font-size:13px">-</span><span
                class=""
                style="font-family:arial,sans-serif;font-size:13px">engine</span><span
                style="font-family:arial,sans-serif;font-size:13px">.</span><span
                style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br
                style="font-family:arial,sans-serif;font-size:13px">
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    Andrew/Jiri,<br>
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Would it be possible to post gluster logs of both the mount
    and bricks on the bz? I can take a look at it once. If I gather
    nothing then probably I will ask for your help in re-creating the
    issue.<br>
    <br>
    Pranith<br>
    <br>
    <blockquote
cite="mid:CAD7dF9dRMRa_hMOg0qvKAjm1LkyuunR5sHZwBizJnscfJwssJQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div>
              <br style="font-family:arial,sans-serif;font-size:13px">
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif"><span
                  style="font-size:13px;font-family:arial,sans-serif">It's
                  definitely connected to the storage which leads us to
                  the&nbsp;</span><span class=""
                  style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
                  style="font-size:13px;font-family:arial,sans-serif">,
                  I'm not very familiar with the&nbsp;</span><span class=""
                  style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
                  style="font-size:13px;font-family:arial,sans-serif">&nbsp;so
                  I need to check this with our&nbsp;</span><span class=""
                  style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
                  style="font-size:13px;font-family:arial,sans-serif">&nbsp;gurus.</span>&#8203;</div>
              <br>
            </div>
            <div>
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif">&#8203;==&#8203;</div>
              <br>
            </div>
            <div>&nbsp;</div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br>
              Vijay<br>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>