<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/19/2014 11:25 AM, Andrew Lau
wrote:<br>
</div>
<blockquote
cite="mid:CAD7dF9coKrJnvd-iAbhTLb0vZsxHcUPGAS1NAa1Bv-+JQZqC2A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Jul 19, 2014 at 12:03 AM,
Pranith Kumar Karampuri <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:pkarampu@redhat.com"
target="_blank">pkarampu@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5"> <br>
<div>On 07/18/2014 05:43 PM, Andrew Lau wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div style="font-family:tahoma,sans-serif"> </div>
<div style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_extra">
<div class="gmail_quote">On Fri, Jul 18, 2014
at 10:06 PM, Vijay Bellur <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:vbellur@redhat.com"
target="_blank">vbellur@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding
gluster-devel]
<div><br>
<br>
On 07/18/2014 05:20 PM, Andrew Lau
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi
all,<br>
<br>
As most of you have got hints from
previous messages, hosted engine<br>
won't work on gluster . A quote from
BZ1097639<br>
<br>
"Using hosted engine with Gluster
backed storage is currently something<br>
we really warn against.<br>
<br>
<br>
I think this bug should be closed or
re-targeted at documentation, because
there is nothing we can do here.
Hosted engine assumes that all writes
are atomic and (immediately) available
for all hosts in the cluster. Gluster
violates those assumptions.<br>
"<br>
</blockquote>
</div>
I tried going through BZ1097639 but could
not find much detail with respect to
gluster there.<br>
<br>
A few questions around the problem:<br>
<br>
1. Can somebody please explain in detail
the scenario that causes the problem?<br>
<br>
2. Is hosted engine performing synchronous
writes to ensure that writes are durable?<br>
<br>
Also, if there is any documentation that
details the hosted engine architecture
that would help in enhancing our
understanding of its interactions with
gluster.
<div><br>
<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<br>
Now my question, does this theory
prevent a scenario of perhaps<br>
something like a gluster replicated
volume being mounted as a glusterfs<br>
filesystem and then re-exported as the
native kernel NFS share for the<br>
hosted-engine to consume? It could
then be possible to chuck ctdb in<br>
there to provide a last resort
failover solution. I have tried myself<br>
and suggested it to two people who are
running a similar setup. Now<br>
using the native kernel NFS server for
hosted-engine and they haven't<br>
reported as many issues. Curious,
could anyone validate my theory on
this?<br>
<br>
</blockquote>
<br>
</div>
If we obtain more details on the use case
and obtain gluster logs from the failed
scenarios, we should be able to understand
the problem better. That could be the
first step in validating your theory or
evolving further recommendations :).<br>
<br>
</blockquote>
<div><br>
</div>
<div>
<div style="font-family:tahoma,sans-serif">
I'm not sure how useful this is, but
Jiri Moskovcak tracked this down in an
off list message.</div>
<br>
</div>
<div>
<div style="font-family:tahoma,sans-serif">
Message Quote:</div>
<br>
</div>
<div>
<div style="font-family:tahoma,sans-serif">
==</div>
<br>
</div>
<div>
<div style="font-family:tahoma,sans-serif">
<span
style="font-family:arial,sans-serif;font-size:13px">We
were able to track it down to this
(thanks Andrew for providing the
testing setup):</span></div>
<br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><span
style="font-family:arial,sans-serif;font-size:13px">h</span><span
style="font-family:arial,sans-serif;font-size:13px">a_agent
service_type=</span><span
style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
style="font-family:arial,sans-serif;font-size:13px">-</span><span
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">'</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">Traceback
(most recent call last):</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",
line 165, in handle</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
response = "success " +
self._dispatch(data)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",
line 261, in _dispatch</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
.get_all_stats_for_service_</span><span
style="font-family:arial,sans-serif;font-size:13px">typ</span><span
style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",
line 41, in
get_all_stats_for_service_type</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
d = self.get_raw_stats_for_</span><span
style="font-family:arial,sans-serif;font-size:13px">service</span><span
style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir,
service_type)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",
line 74, in
get_raw_stats_for_service_type</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
f = os.open(path, direct_flag |
os.O_RDONLY)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">OSError:
[Errno 116] Stale file handle:
'/rhev/data-center/mnt/</span><span
style="font-family:arial,sans-serif;font-size:13px">localho</span><span
style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">/</span><span
style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span
style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><span
style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span
style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span
style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
style="font-family:arial,sans-serif;font-size:13px">-</span><span
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">.</span><span
style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br
style="font-family:arial,sans-serif;font-size:13px">
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
Andrew/Jiri,<br>
Would it be possible to post gluster logs of
both the mount and bricks on the bz? I can take a look
at it once. If I gather nothing then probably I will ask
for your help in re-creating the issue.<br>
<br>
Pranith<br>
</div>
</blockquote>
<div><br>
</div>
<div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif">Unfortunately, I
don't have the logs for that setup any more.. I'll try
replicate when I get a chance. If I understand the
comment from the BZ, I don't think it's a gluster bug
per-say, more just how gluster does its replication.</div>
</div>
</div>
</div>
</div>
</blockquote>
hi Andrew,<br>
Thanks for that. I couldn't come to any conclusions because
no logs were available. It is unlikely that self-heal is involved
because there were no bricks going down/up according to the bug
description.<br>
<br>
Pranith<br>
<blockquote
cite="mid:CAD7dF9coKrJnvd-iAbhTLb0vZsxHcUPGAS1NAa1Bv-+JQZqC2A@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> <br
style="font-family:arial,sans-serif;font-size:13px">
<div style="font-family:tahoma,sans-serif"><span
style="font-size:13px;font-family:arial,sans-serif">It's definitely
connected to the storage which leads us to
the </span><span
style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif">, I'm not very
familiar with the </span><span
style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif"> so I need to check
this with our </span><span
style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif"> gurus.</span></div>
<br>
</div>
<div>
<div style="font-family:tahoma,sans-serif">==
</div>
<br>
</div>
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br>
Vijay<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-devel mailing list
<a moz-do-not-send="true" href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>