5:03 p.m.
This is a multi-part message in MIME format.
--------------010502030401070306090405
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur(a)redhat.com
<mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted
engine
won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently
something
we really warn against.
I think this bug should be closed or re-targeted at
documentation, because there is nothing we can do here. Hosted
engine assumes that all writes are atomic and (immediately)
available for all hosts in the cluster. Gluster violates those
assumptions.
"
I tried going through BZ1097639 but could not find much detail
with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes
the problem?
2. Is hosted engine performing synchronous writes to ensure that
writes are durable?
Also, if there is any documentation that details the hosted engine
architecture that would help in enhancing our understanding of its
interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps
something like a gluster replicated volume being mounted as a
glusterfs
filesystem and then re-exported as the native kernel NFS share
for the
hosted-engine to consume? It could then be possible to chuck
ctdb in
there to provide a last resort failover solution. I have tried
myself
and suggested it to two people who are running a similar
setup. Now
using the native kernel NFS server for hosted-engine and they
haven't
reported as many issues. Curious, could anyone validate my
theory on this?
If we obtain more details on the use case and obtain gluster logs
from the failed scenarios, we should be able to understand the
problem better. That could be the first step in validating your
theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down
in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the
testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine'
Traceback (most recent call last):
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 165, in handle
response = "success " + self._dispatch(data)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 261, in _dispatch
.get_all_stats_for_service_type(**options)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 41, in get_all_stats_for_service_type
d = self.get_raw_stats_for_service_type(storage_dir, service_type)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 74, in get_raw_stats_for_service_type
f = os.open(path, direct_flag | os.O_RDONLY)
OSError: [Errno 116] Stale file handle:
'/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri,
Would it be possible to post gluster logs of both the mount and
bricks on the bz? I can take a look at it once. If I gather nothing then
probably I will ask for your help in re-creating the issue.
Pranith
It's definitely connected to the storage which leads us to the
gluster, I'm not very familiar with the gluster so I need to check
this with our gluster gurus.
==
Thanks,
Vijay
_______________________________________________
Gluster-devel mailing list
Gluster-devel(a)gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
--------------010502030401070306090405
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/18/2014 05:43 PM, Andrew Lau
wrote:<br>
</div>
<blockquote
cite="mid:CAD7dF9dRMRa_hMOg0qvKAjm1LkyuunR5sHZwBizJnscfJwssJQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default"
style="font-family:tahoma,sans-serif">​
​</div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif"><br>
</div>
<div class="gmail_extra">
<div class="gmail_quote">On Fri, Jul 18, 2014 at 10:06 PM,
Vijay Bellur <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:vbellur@redhat.com"
target="_blank">vbellur(a)redhat.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding
gluster-devel]
<div class=""><br>
<br>
On 07/18/2014 05:20 PM, Andrew Lau wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi
all,<br>
<br>
As most of you have got hints from previous messages,
hosted engine<br>
won't work on gluster . A quote from BZ1097639<br>
<br>
"Using hosted engine with Gluster backed storage is
currently something<br>
we really warn against.<br>
<br>
<br>
I think this bug should be closed or re-targeted at
documentation, because there is nothing we can do
here. Hosted engine assumes that all writes are atomic
and (immediately) available for all hosts in the
cluster. Gluster violates those assumptions.<br>
​"<br>
</blockquote>
</div>
I tried going through BZ1097639 but could not find much
detail with respect to gluster there.<br>
<br>
A few questions around the problem:<br>
<br>
1. Can somebody please explain in detail the scenario that
causes the problem?<br>
<br>
2. Is hosted engine performing synchronous writes to
ensure that writes are durable?<br>
<br>
Also, if there is any documentation that details the
hosted engine architecture that would help in enhancing
our understanding of its interactions with gluster.
<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">​<br>
<br>
Now my question, does this theory prevent a scenario
of perhaps<br>
something like a gluster replicated volume being
mounted as a glusterfs<br>
filesystem and then re-exported as the native kernel
NFS share for the<br>
hosted-engine to consume? It could then be possible to
chuck ctdb in<br>
there to provide a last resort failover solution. I
have tried myself<br>
and suggested it to two people who are running a
similar setup. Now<br>
using the native kernel NFS server for hosted-engine
and they haven't<br>
reported as many issues. Curious, could anyone
validate my theory on this?<br>
<br>
</blockquote>
<br>
</div>
If we obtain more details on the use case and obtain
gluster logs from the failed scenarios, we should be able
to understand the problem better. That could be the first
step in validating your theory or evolving further
recommendations :).<br>
<br>
</blockquote>
<div><br>
</div>
<div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif">​I'm not
sure how
useful this is, but ​Jiri Moskovcak tracked this down in
an off list message.</div>
<br>
</div>
<div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif">​Message
Quote:​</div>
<br>
</div>
<div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif">​==​</div>
<br>
</div>
<div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif">
​<span
style="font-family:arial,sans-serif;font-size:13px">We
were able to track it down to this (thanks Andrew for
providing the testing setup):</span></div>
<br style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><span
style="font-family:arial,sans-serif;font-size:13px">h</span><span
style="font-family:arial,sans-serif;font-size:13px">a_agent
service_type=</span><span class=""
style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
style="font-family:arial,sans-serif;font-size:13px">-</span><span
class=""
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">'</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">Traceback
(most recent call last):</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",
line 165, in handle</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
response = "success " +
self._dispatch(data)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/listener.py",
line 261, in _dispatch</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
.get_all_stats_for_service_</span><span
style="font-family:arial,sans-serif;font-size:13px">typ</span><span
style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",
line 41, in get_all_stats_for_service_type</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
d = self.get_raw_stats_for_</span><span
style="font-family:arial,sans-serif;font-size:13px">service</span><span
style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir,
service_type)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
File "/usr/lib/python2.6/site-</span><span
style="font-family:arial,sans-serif;font-size:13px">packa</span><span
style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span
style="font-family:arial,sans-serif;font-size:13px">ha/</span><span
style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py",
line 74, in get_raw_stats_for_service_type</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">
f = os.open(path, direct_flag |
os.O_RDONLY)</span><br
style="font-family:arial,sans-serif;font-size:13px">
<span
style="font-family:arial,sans-serif;font-size:13px">OSError:
[Errno 116] Stale file handle:
'/rhev/data-center/mnt/</span><span
style="font-family:arial,sans-serif;font-size:13px">localho</span><span
style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span
class=""
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">/</span><span
style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span
style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><span
style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span
style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span
class=""
style="font-family:arial,sans-serif;font-size:13px">hosted</span><span
style="font-family:arial,sans-serif;font-size:13px">-</span><span
class=""
style="font-family:arial,sans-serif;font-size:13px">engine</span><span
style="font-family:arial,sans-serif;font-size:13px">.</span><span
style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br
style="font-family:arial,sans-serif;font-size:13px">
</div>
</div>
</div>
</div>
</blockquote>
Andrew/Jiri,<br>
Would it be
possible to post gluster logs of both the mount
and bricks on the bz? I can take a look at it once. If I gather
nothing then probably I will ask for your help in re-creating the
issue.<br>
<br>
Pranith<br>
<br>
<blockquote
cite="mid:CAD7dF9dRMRa_hMOg0qvKAjm1LkyuunR5sHZwBizJnscfJwssJQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<br style="font-family:arial,sans-serif;font-size:13px">
<div class="gmail_default"
style="font-family:tahoma,sans-serif"><span
style="font-size:13px;font-family:arial,sans-serif">It's
definitely connected to the storage which leads us to
the </span><span class=""
style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif">,
I'm not very familiar with the </span><span
class=""
style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif"> so
I need to check this with our </span><span
class=""
style="font-size:13px;font-family:arial,sans-serif">gluster</span><span
style="font-size:13px;font-family:arial,sans-serif"> gurus.</span>​</div>
<br>
</div>
<div>
<div class="gmail_default"
style="font-family:tahoma,sans-serif">​==​</div>
<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br>
Vijay<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel&...
</pre>
</blockquote>
<br>
</body>
</html>
--------------010502030401070306090405--