<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <br>
    <div class="moz-cite-prefix">On 07/19/2014 11:25 AM, Andrew Lau
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAD7dF9coKrJnvd-iAbhTLb0vZsxHcUPGAS1NAa1Bv-+JQZqC2A@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
        </div>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Sat, Jul 19, 2014 at 12:03 AM,
            Pranith Kumar Karampuri <span dir="ltr">&lt;<a
                moz-do-not-send="true" href="mailto:pkarampu@redhat.com"
                target="_blank">pkarampu@redhat.com</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF">
                <div>
                  <div class="h5"> <br>
                    <div>On 07/18/2014 05:43 PM, Andrew Lau wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div style="font-family:tahoma,sans-serif">​ ​</div>
                        <div style="font-family:tahoma,sans-serif"><br>
                        </div>
                        <div class="gmail_extra">
                          <div class="gmail_quote">On Fri, Jul 18, 2014
                            at 10:06 PM, Vijay Bellur <span dir="ltr">&lt;<a
                                moz-do-not-send="true"
                                href="mailto:vbellur@redhat.com"
                                target="_blank">vbellur@redhat.com</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote"
                              style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding

                              gluster-devel]
                              <div><br>
                                <br>
                                On 07/18/2014 05:20 PM, Andrew Lau
                                wrote:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi

                                  all,<br>
                                  <br>
                                  As most of you have got hints from
                                  previous messages, hosted engine<br>
                                  won't work on gluster . A quote from
                                  BZ1097639<br>
                                  <br>
                                  "Using hosted engine with Gluster
                                  backed storage is currently something<br>
                                  we really warn against.<br>
                                  <br>
                                  <br>
                                  I think this bug should be closed or
                                  re-targeted at documentation, because
                                  there is nothing we can do here.
                                  Hosted engine assumes that all writes
                                  are atomic and (immediately) available
                                  for all hosts in the cluster. Gluster
                                  violates those assumptions.<br>
                                  ​"<br>
                                </blockquote>
                              </div>
                              I tried going through BZ1097639 but could
                              not find much detail with respect to
                              gluster there.<br>
                              <br>
                              A few questions around the problem:<br>
                              <br>
                              1. Can somebody please explain in detail
                              the scenario that causes the problem?<br>
                              <br>
                              2. Is hosted engine performing synchronous
                              writes to ensure that writes are durable?<br>
                              <br>
                              Also, if there is any documentation that
                              details the hosted engine architecture
                              that would help in enhancing our
                              understanding of its interactions with
                              gluster.
                              <div><br>
                                <br>
                                <blockquote class="gmail_quote"
                                  style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">​<br>
                                  <br>
                                  Now my question, does this theory
                                  prevent a scenario of perhaps<br>
                                  something like a gluster replicated
                                  volume being mounted as a glusterfs<br>
                                  filesystem and then re-exported as the
                                  native kernel NFS share for the<br>
                                  hosted-engine to consume? It could
                                  then be possible to chuck ctdb in<br>
                                  there to provide a last resort
                                  failover solution. I have tried myself<br>
                                  and suggested it to two people who are
                                  running a similar setup. Now<br>
                                  using the native kernel NFS server for
                                  hosted-engine and they haven't<br>
                                  reported as many issues. Curious,
                                  could anyone validate my theory on
                                  this?<br>
                                  <br>
                                </blockquote>
                                <br>
                              </div>
                              If we obtain more details on the use case
                              and obtain gluster logs from the failed
                              scenarios, we should be able to understand
                              the problem better. That could be the
                              first step in validating your theory or
                              evolving further recommendations :).<br>
                              <br>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>
                              <div style="font-family:tahoma,sans-serif">​
                                I'm not sure how useful this is, but
                                ​Jiri Moskovcak tracked this down in an
                                off list message.</div>
                              <br>
                            </div>
                            <div>
                              <div style="font-family:tahoma,sans-serif">​
                                Message Quote:​</div>
                              <br>
                            </div>
                            <div>
                              <div style="font-family:tahoma,sans-serif">​
                                ==​</div>
                              <br>
                            </div>
                            <div>
                              <div style="font-family:tahoma,sans-serif">
                                ​<span
                                  style="font-family:arial,sans-serif;font-size:13px">We

                                  were able to track it down to this
                                  (thanks Andrew for providing the
                                  testing setup):</span></div>
                              <br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><span
style="font-family:arial,sans-serif;font-size:13px">h</span><span
                                style="font-family:arial,sans-serif;font-size:13px">a_agent

                                service_type=</span><span
                                style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
style="font-family:arial,sans-serif;font-size:13px">-</span><span
                                style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">'</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px">Traceback

                                (most recent call last):</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                                style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",

                                line 165, in handle</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                  response = "success " +
                                self._dispatch(data)</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                                style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",

                                line 261, in _dispatch</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                  .get_all_stats_for_service_</span><span
style="font-family:arial,sans-serif;font-size:13px">typ</span><span
                                style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br
style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                                style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",

                                line 41, in
                                get_all_stats_for_service_type</span><br
style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                  d = self.get_raw_stats_for_</span><span
style="font-family:arial,sans-serif;font-size:13px">service</span><span
style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir,
                                service_type)</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
                                style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
                                style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",

                                line 74, in
                                get_raw_stats_for_service_type</span><br
style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px"> 
                                  f = os.open(path, direct_flag |
                                os.O_RDONLY)</span><br
                                style="font-family:arial,sans-serif;font-size:13px">
                              <span
                                style="font-family:arial,sans-serif;font-size:13px">OSError:

                                [Errno 116] Stale file handle:
                                '/rhev/data-center/mnt/</span><span
                                style="font-family:arial,sans-serif;font-size:13px">localho</span><span
style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
                                style="font-family:arial,sans-serif;font-size:13px">/</span><span
style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span
style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><span
style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span
style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span
style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
                                style="font-family:arial,sans-serif;font-size:13px">-</span><span
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
                                style="font-family:arial,sans-serif;font-size:13px">.</span><span
style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br
style="font-family:arial,sans-serif;font-size:13px">
                            </div>
                          </div>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                </div>
                Andrew/Jiri,<br>
                        Would it be possible to post gluster logs of
                both the mount and bricks on the bz? I can take a look
                at it once. If I gather nothing then probably I will ask
                for your help in re-creating the issue.<br>
                <br>
                Pranith<br>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>
              <div class="gmail_default"
                style="font-family:tahoma,sans-serif">​Unfortunately, I
                don't have the logs for that setup any more.. ​I'll try
                replicate when I get a chance. If I understand the
                comment from the BZ, I don't think it's a gluster bug
                per-say, more just how gluster does its replication.</div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    hi Andrew,<br>
             Thanks for that. I couldn't come to any conclusions because
    no logs were available. It is unlikely that self-heal is involved
    because there were no bricks going down/up according to the bug
    description.<br>
    <br>
    Pranith<br>
    <blockquote
cite="mid:CAD7dF9coKrJnvd-iAbhTLb0vZsxHcUPGAS1NAa1Bv-+JQZqC2A@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div>
              <br>
            </div>
            <div> </div>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> <br>
                <blockquote type="cite">
                  <div dir="ltr">
                    <div class="gmail_extra">
                      <div class="gmail_quote">
                        <div> <br
                            style="font-family:arial,sans-serif;font-size:13px">
                          <div style="font-family:tahoma,sans-serif"><span
style="font-size:13px;font-family:arial,sans-serif">It's definitely
                              connected to the storage which leads us to
                              the </span><span
                              style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif">, I'm not very
                              familiar with the </span><span
                              style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif"> so I need to check
                              this with our </span><span
                              style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif"> gurus.</span>​</div>
                          <br>
                        </div>
                        <div>
                          <div style="font-family:tahoma,sans-serif">​==
                            ​</div>
                          <br>
                        </div>
                        <div> </div>
                        <blockquote class="gmail_quote"
                          style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br>
                          Vijay<br>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </div>
                  <br>
                  <fieldset></fieldset>
                  <br>
                  <pre>_______________________________________________
Gluster-devel mailing list
<a moz-do-not-send="true" href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
                </blockquote>
                <br>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>