Can we debug some truths/myths/facts about hosted-engine and gluster?
Hi all, As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639 "Using hosted engine with Gluster backed storage is currently something we really warn against. I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. " Until the documentation gets updated, I hope this serves as a useful notice at least to save people some of the headaches I hit like hosted-engine starting up multiple VMs because of above issue. Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this? Thanks, Andrew
[Adding gluster-devel] On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. " I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem: 1. Can somebody please explain in detail the scenario that causes the problem? 2. Is hosted engine performing synchronous writes to ensure that writes are durable? Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :). Thanks, Vijay
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message. Message Quote: == We were able to track it down to this (thanks Andrew for providing the testing setup): -b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localho st:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted- engine.metadata' It's definitely connected to the storage which leads us to the gluster, I'm not very familiar with the gluster so I need to check this with our gluster gurus. ==
Thanks, Vijay
This is a multi-part message in MIME format. --------------010502030401070306090405 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue. Pranith
It's definitely connected to the storage which leads us to the gluster, I'm not very familiar with the gluster so I need to check this with our gluster gurus.
==
Thanks, Vijay
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
--------------010502030401070306090405 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 07/18/2014 05:43 PM, Andrew Lau wrote:<br> </div> <blockquote cite="mid:CAD7dF9dRMRa_hMOg0qvKAjm1LkyuunR5sHZwBizJnscfJwssJQ@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif"> </div> <div class="gmail_default" style="font-family:tahoma,sans-serif"><br> </div> <div class="gmail_extra"> <div class="gmail_quote">On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <span dir="ltr"><<a moz-do-not-send="true" href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding gluster-devel] <div class=""><br> <br> On 07/18/2014 05:20 PM, Andrew Lau wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi all,<br> <br> As most of you have got hints from previous messages, hosted engine<br> won't work on gluster . A quote from BZ1097639<br> <br> "Using hosted engine with Gluster backed storage is currently something<br> we really warn against.<br> <br> <br> I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions.<br> "<br> </blockquote> </div> I tried going through BZ1097639 but could not find much detail with respect to gluster there.<br> <br> A few questions around the problem:<br> <br> 1. Can somebody please explain in detail the scenario that causes the problem?<br> <br> 2. Is hosted engine performing synchronous writes to ensure that writes are durable?<br> <br> Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster. <div class=""><br> <br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br> <br> Now my question, does this theory prevent a scenario of perhaps<br> something like a gluster replicated volume being mounted as a glusterfs<br> filesystem and then re-exported as the native kernel NFS share for the<br> hosted-engine to consume? It could then be possible to chuck ctdb in<br> there to provide a last resort failover solution. I have tried myself<br> and suggested it to two people who are running a similar setup. Now<br> using the native kernel NFS server for hosted-engine and they haven't<br> reported as many issues. Curious, could anyone validate my theory on this?<br> <br> </blockquote> <br> </div> If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).<br> <br> </blockquote> <div><br> </div> <div> <div class="gmail_default" style="font-family:tahoma,sans-serif">I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.</div> <br> </div> <div> <div class="gmail_default" style="font-family:tahoma,sans-serif">Message Quote:</div> <br> </div> <div> <div class="gmail_default" style="font-family:tahoma,sans-serif">==</div> <br> </div> <div> <div class="gmail_default" style="font-family:tahoma,sans-serif"> <span style="font-family:arial,sans-serif;font-size:13px">We were able to track it down to this (thanks Andrew for providing the testing setup):</span></div> <br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><span style="font-family:arial,sans-serif;font-size:13px">h</span><span style="font-family:arial,sans-serif;font-size:13px">a_agent service_type=</span><span class="" style="font-family:arial,sans-serif;font-size:13px">hosted</span><span style="font-family:arial,sans-serif;font-size:13px">-</span><span class="" style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">'</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px">Traceback (most recent call last):</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/listener.py", line 165, in handle</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> response = "success " + self._dispatch(data)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/listener.py", line 261, in _dispatch</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> .get_all_stats_for_service_</span><span style="font-family:arial,sans-serif;font-size:13px">typ</span><span style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py", line 41, in get_all_stats_for_service_type</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> d = self.get_raw_stats_for_</span><span style="font-family:arial,sans-serif;font-size:13px">service</span><span style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir, service_type)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py", line 74, in get_raw_stats_for_service_type</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> f = os.open(path, direct_flag | os.O_RDONLY)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px">OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/</span><span style="font-family:arial,sans-serif;font-size:13px">localho</span><span style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span class="" style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">/</span><span style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><span style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span class="" style="font-family:arial,sans-serif;font-size:13px">hosted</span><span style="font-family:arial,sans-serif;font-size:13px">-</span><span class="" style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">.</span><span style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br style="font-family:arial,sans-serif;font-size:13px"> </div> </div> </div> </div> </blockquote> Andrew/Jiri,<br> Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.<br> <br> Pranith<br> <br> <blockquote cite="mid:CAD7dF9dRMRa_hMOg0qvKAjm1LkyuunR5sHZwBizJnscfJwssJQ@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <br style="font-family:arial,sans-serif;font-size:13px"> <div class="gmail_default" style="font-family:tahoma,sans-serif"><span style="font-size:13px;font-family:arial,sans-serif">It's definitely connected to the storage which leads us to the </span><span class="" style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif">, I'm not very familiar with the </span><span class="" style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif"> so I need to check this with our </span><span class="" style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif"> gurus.</span></div> <br> </div> <div> <div class="gmail_default" style="font-family:tahoma,sans-serif">==</div> <br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br> Vijay<br> </blockquote> </div> <br> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Gluster-devel mailing list <a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a> <a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a> </pre> </blockquote> <br> </body> </html> --------------010502030401070306090405--
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri < pkarampu@redhat.com> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localho st:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted -engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
It's definitely connected to the storage which leads us to the gluster, I'm not very familiar with the gluster so I need to check this with our gluster gurus.
==
Thanks, Vijay
_______________________________________________ Gluster-devel mailing listGluster-devel@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-devel
This is a multi-part message in MIME format. --------------080406000000050709070606 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 07/19/2014 11:25 AM, Andrew Lau wrote:
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <pkarampu@redhat.com <mailto:pkarampu@redhat.com>> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
hi Andrew, Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description. Pranith
It's definitely connected to the storage which leads us to the gluster, I'm not very familiar with the gluster so I need to check this with our gluster gurus.
==
Thanks, Vijay
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
--------------080406000000050709070606 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=UTF-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <div class="moz-cite-prefix">On 07/19/2014 11:25 AM, Andrew Lau wrote:<br> </div> <blockquote cite="mid:CAD7dF9coKrJnvd-iAbhTLb0vZsxHcUPGAS1NAa1Bv-+JQZqC2A@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_default" style="font-family:tahoma,sans-serif"><br> </div> <div class="gmail_extra"><br> <div class="gmail_quote">On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <span dir="ltr"><<a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000000" bgcolor="#FFFFFF"> <div> <div class="h5"> <br> <div>On 07/18/2014 05:43 PM, Andrew Lau wrote:<br> </div> <blockquote type="cite"> <div dir="ltr"> <div style="font-family:tahoma,sans-serif"> </div> <div style="font-family:tahoma,sans-serif"><br> </div> <div class="gmail_extra"> <div class="gmail_quote">On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <span dir="ltr"><<a moz-do-not-send="true" href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding gluster-devel] <div><br> <br> On 07/18/2014 05:20 PM, Andrew Lau wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi all,<br> <br> As most of you have got hints from previous messages, hosted engine<br> won't work on gluster . A quote from BZ1097639<br> <br> "Using hosted engine with Gluster backed storage is currently something<br> we really warn against.<br> <br> <br> I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions.<br> "<br> </blockquote> </div> I tried going through BZ1097639 but could not find much detail with respect to gluster there.<br> <br> A few questions around the problem:<br> <br> 1. Can somebody please explain in detail the scenario that causes the problem?<br> <br> 2. Is hosted engine performing synchronous writes to ensure that writes are durable?<br> <br> Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster. <div><br> <br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br> <br> Now my question, does this theory prevent a scenario of perhaps<br> something like a gluster replicated volume being mounted as a glusterfs<br> filesystem and then re-exported as the native kernel NFS share for the<br> hosted-engine to consume? It could then be possible to chuck ctdb in<br> there to provide a last resort failover solution. I have tried myself<br> and suggested it to two people who are running a similar setup. Now<br> using the native kernel NFS server for hosted-engine and they haven't<br> reported as many issues. Curious, could anyone validate my theory on this?<br> <br> </blockquote> <br> </div> If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).<br> <br> </blockquote> <div><br> </div> <div> <div style="font-family:tahoma,sans-serif"> I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.</div> <br> </div> <div> <div style="font-family:tahoma,sans-serif"> Message Quote:</div> <br> </div> <div> <div style="font-family:tahoma,sans-serif"> ==</div> <br> </div> <div> <div style="font-family:tahoma,sans-serif"> <span style="font-family:arial,sans-serif;font-size:13px">We were able to track it down to this (thanks Andrew for providing the testing setup):</span></div> <br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><span style="font-family:arial,sans-serif;font-size:13px">h</span><span style="font-family:arial,sans-serif;font-size:13px">a_agent service_type=</span><span style="font-family:arial,sans-serif;font-size:13px">hosted</span><span style="font-family:arial,sans-serif;font-size:13px">-</span><span style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">'</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px">Traceback (most recent call last):</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/listener.py", line 165, in handle</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> response = "success " + self._dispatch(data)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/listener.py", line 261, in _dispatch</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> .get_all_stats_for_service_</span><span style="font-family:arial,sans-serif;font-size:13px">typ</span><span style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py", line 41, in get_all_stats_for_service_type</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> d = self.get_raw_stats_for_</span><span style="font-family:arial,sans-serif;font-size:13px">service</span><span style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir, service_type)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py", line 74, in get_raw_stats_for_service_type</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px"> f = os.open(path, direct_flag | os.O_RDONLY)</span><br style="font-family:arial,sans-serif;font-size:13px"> <span style="font-family:arial,sans-serif;font-size:13px">OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/</span><span style="font-family:arial,sans-serif;font-size:13px">localho</span><span style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">/</span><span style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><span style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span style="font-family:arial,sans-serif;font-size:13px">hosted</span><span style="font-family:arial,sans-serif;font-size:13px">-</span><span style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">.</span><span style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br style="font-family:arial,sans-serif;font-size:13px"> </div> </div> </div> </div> </blockquote> </div> </div> Andrew/Jiri,<br> Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.<br> <br> Pranith<br> </div> </blockquote> <div><br> </div> <div> <div class="gmail_default" style="font-family:tahoma,sans-serif">Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.</div> </div> </div> </div> </div> </blockquote> hi Andrew,<br> Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description.<br> <br> Pranith<br> <blockquote cite="mid:CAD7dF9coKrJnvd-iAbhTLb0vZsxHcUPGAS1NAa1Bv-+JQZqC2A@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000000" bgcolor="#FFFFFF"> <br> <blockquote type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <div> <br style="font-family:arial,sans-serif;font-size:13px"> <div style="font-family:tahoma,sans-serif"><span style="font-size:13px;font-family:arial,sans-serif">It's definitely connected to the storage which leads us to the </span><span style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif">, I'm not very familiar with the </span><span style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif"> so I need to check this with our </span><span style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif"> gurus.</span></div> <br> </div> <div> <div style="font-family:tahoma,sans-serif">== </div> <br> </div> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks,<br> Vijay<br> </blockquote> </div> <br> </div> </div> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Gluster-devel mailing list <a moz-do-not-send="true" href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> <a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a> </pre> </blockquote> <br> </div> </blockquote> </div> <br> </div> </div> </blockquote> <br> </body> </html> --------------080406000000050709070606--
On 07/19/2014 08:58 AM, Pranith Kumar Karampuri wrote:
On 07/19/2014 11:25 AM, Andrew Lau wrote:
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <pkarampu@redhat.com <mailto:pkarampu@redhat.com>> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
hi Andrew, Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description.
Hi, I've never had such setup, I guessed problem with gluster based on "OSError: [Errno 116] Stale file handle:" which happens when the file opened by application on client gets removed on the server. I'm pretty sure we (hosted-engine) don't remove that file, so I think it's some gluster magic moving the data around... --Jirka
Pranith
It's definitely connected to the storage which leads us to the gluster, I'm not very familiar with the gluster so I need to check this with our gluster gurus.
==
Thanks, Vijay
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
On 07/21/2014 02:08 PM, Jiri Moskovcak wrote:
On 07/19/2014 08:58 AM, Pranith Kumar Karampuri wrote:
On 07/19/2014 11:25 AM, Andrew Lau wrote:
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <pkarampu@redhat.com <mailto:pkarampu@redhat.com>> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
hi Andrew, Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description.
Hi, I've never had such setup, I guessed problem with gluster based on "OSError: [Errno 116] Stale file handle:" which happens when the file opened by application on client gets removed on the server. I'm pretty sure we (hosted-engine) don't remove that file, so I think it's some gluster magic moving the data around... Hi, Without bricks going up/down or there are new bricks added data is not moved around by gluster unless a file operation comes to gluster. So I am still not sure why this happened.
Pranith
--Jirka
Pranith
It's definitely connected to the storage which leads us to the gluster, I'm not very familiar with the gluster so I need to check this with our gluster gurus.
==
Thanks, Vijay
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
On 07/21/2014 05:09 AM, Pranith Kumar Karampuri wrote:
On 07/21/2014 02:08 PM, Jiri Moskovcak wrote:
On 07/19/2014 08:58 AM, Pranith Kumar Karampuri wrote:
On 07/19/2014 11:25 AM, Andrew Lau wrote:
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <pkarampu@redhat.com <mailto:pkarampu@redhat.com>> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
hi Andrew, Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description.
Hi, I've never had such setup, I guessed problem with gluster based on "OSError: [Errno 116] Stale file handle:" which happens when the file opened by application on client gets removed on the server. I'm pretty sure we (hosted-engine) don't remove that file, so I think it's some gluster magic moving the data around... Hi, Without bricks going up/down or there are new bricks added data is not moved around by gluster unless a file operation comes to gluster. So I am still not sure why this happened.
Does hosted engine perform deletion & re-creation of file <uuid>/ha_agent/hosted-engine.metadata in some operational sequence? In such a case, if this file is accessed by a stale gfid, ESTALE is possible. I see references to 2 hosted engines being operational in the bug report and that makes me wonder if this is a likely scenario? I am also curious to understand why NFS was chosen as the access method to the gluster volume. Isn't FUSE based access a possibility here? -Vijay
On 07/22/2014 04:28 AM, Vijay Bellur wrote:
On 07/21/2014 05:09 AM, Pranith Kumar Karampuri wrote:
On 07/21/2014 02:08 PM, Jiri Moskovcak wrote:
On 07/19/2014 08:58 AM, Pranith Kumar Karampuri wrote:
On 07/19/2014 11:25 AM, Andrew Lau wrote:
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <pkarampu@redhat.com <mailto:pkarampu@redhat.com>> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote:
On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:
[Adding gluster-devel]
On 07/18/2014 05:20 PM, Andrew Lau wrote:
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions. "
I tried going through BZ1097639 but could not find much detail with respect to gluster there.
A few questions around the problem:
1. Can somebody please explain in detail the scenario that causes the problem?
2. Is hosted engine performing synchronous writes to ensure that writes are durable?
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).
I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.
Message Quote:
==
We were able to track it down to this (thanks Andrew for providing the testing setup):
-b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine' Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 165, in handle response = "success " + self._dispatch(data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
line 261, in _dispatch .get_all_stats_for_service_type(**options) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 41, in get_all_stats_for_service_type d = self.get_raw_stats_for_service_type(storage_dir, service_type) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 74, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY) OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
hi Andrew, Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description.
Hi, I've never had such setup, I guessed problem with gluster based on "OSError: [Errno 116] Stale file handle:" which happens when the file opened by application on client gets removed on the server. I'm pretty sure we (hosted-engine) don't remove that file, so I think it's some gluster magic moving the data around... Hi, Without bricks going up/down or there are new bricks added data is not moved around by gluster unless a file operation comes to gluster. So I am still not sure why this happened.
Does hosted engine perform deletion & re-creation of file <uuid>/ha_agent/hosted-engine.metadata in some operational sequence? In such a case, if this file is accessed by a stale gfid, ESTALE is possible.
I see references to 2 hosted engines being operational in the bug report and that makes me wonder if this is a likely scenario?
I am also curious to understand why NFS was chosen as the access method to the gluster volume. Isn't FUSE based access a possibility here?
it is, but it wasn't enabled in the setup due to multiple reports around gluster robustness with sanlock at the time. iiuc, with replica 3 we should be in a much better place and re-enable it (also validating its replica 3 probably?)
On 07/22/2014 07:21 AM, Itamar Heim wrote:
On 07/22/2014 04:28 AM, Vijay Bellur wrote:
On 07/21/2014 05:09 AM, Pranith Kumar Karampuri wrote:
On 07/21/2014 02:08 PM, Jiri Moskovcak wrote:
On 07/19/2014 08:58 AM, Pranith Kumar Karampuri wrote:
On 07/19/2014 11:25 AM, Andrew Lau wrote:
On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri <pkarampu@redhat.com <mailto:pkarampu@redhat.com>> wrote:
On 07/18/2014 05:43 PM, Andrew Lau wrote: > > > On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur > <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote: > > [Adding gluster-devel] > > > On 07/18/2014 05:20 PM, Andrew Lau wrote: > > Hi all, > > As most of you have got hints from previous messages, > hosted engine > won't work on gluster . A quote from BZ1097639 > > "Using hosted engine with Gluster backed storage is > currently something > we really warn against. > > > I think this bug should be closed or re-targeted at > documentation, because there is nothing we can do here. > Hosted engine assumes that all writes are atomic and > (immediately) available for all hosts in the cluster. > Gluster violates those assumptions. > " > > I tried going through BZ1097639 but could not find much > detail with respect to gluster there. > > A few questions around the problem: > > 1. Can somebody please explain in detail the scenario that > causes the problem? > > 2. Is hosted engine performing synchronous writes to ensure > that writes are durable? > > Also, if there is any documentation that details the hosted > engine architecture that would help in enhancing our > understanding of its interactions with gluster. > > > > > Now my question, does this theory prevent a scenario of > perhaps > something like a gluster replicated volume being mounted > as a glusterfs > filesystem and then re-exported as the native kernel NFS > share for the > hosted-engine to consume? It could then be possible to > chuck ctdb in > there to provide a last resort failover solution. I have > tried myself > and suggested it to two people who are running a similar > setup. Now > using the native kernel NFS server for hosted-engine and > they haven't > reported as many issues. Curious, could anyone validate > my theory on this? > > > If we obtain more details on the use case and obtain gluster > logs from the failed scenarios, we should be able to > understand the problem better. That could be the first step > in validating your theory or evolving further > recommendations :). > > > I'm not sure how useful this is, but Jiri Moskovcak tracked > this down in an off list message. > > Message Quote: > > == > > We were able to track it down to this (thanks Andrew for > providing the testing setup): > > -b686-4363-bb7e-dba99e5789b6/ha_agent > service_type=hosted-engine' > Traceback (most recent call last): > File > "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > > > > line 165, in handle > response = "success " + self._dispatch(data) > File > "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", > > > > line 261, in _dispatch > .get_all_stats_for_service_type(**options) > File > "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > > > > line 41, in get_all_stats_for_service_type > d = self.get_raw_stats_for_service_type(storage_dir, > service_type) > File > "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", > > > > line 74, in get_raw_stats_for_service_type > f = os.open(path, direct_flag | os.O_RDONLY) > OSError: [Errno 116] Stale file handle: > '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata' > > > Andrew/Jiri, Would it be possible to post gluster logs of both the mount and bricks on the bz? I can take a look at it once. If I gather nothing then probably I will ask for your help in re-creating the issue.
Pranith
Unfortunately, I don't have the logs for that setup any more.. I'll try replicate when I get a chance. If I understand the comment from the BZ, I don't think it's a gluster bug per-say, more just how gluster does its replication.
hi Andrew, Thanks for that. I couldn't come to any conclusions because no logs were available. It is unlikely that self-heal is involved because there were no bricks going down/up according to the bug description.
Hi, I've never had such setup, I guessed problem with gluster based on "OSError: [Errno 116] Stale file handle:" which happens when the file opened by application on client gets removed on the server. I'm pretty sure we (hosted-engine) don't remove that file, so I think it's some gluster magic moving the data around... Hi, Without bricks going up/down or there are new bricks added data is not moved around by gluster unless a file operation comes to gluster. So I am still not sure why this happened.
Does hosted engine perform deletion & re-creation of file <uuid>/ha_agent/hosted-engine.metadata in some operational sequence? In such a case, if this file is accessed by a stale gfid, ESTALE is possible.
I see references to 2 hosted engines being operational in the bug report and that makes me wonder if this is a likely scenario?
I am also curious to understand why NFS was chosen as the access method to the gluster volume. Isn't FUSE based access a possibility here?
it is, but it wasn't enabled in the setup due to multiple reports around gluster robustness with sanlock at the time. iiuc, with replica 3 we should be in a much better place and re-enable it (also validating its replica 3 probably?)
Yes, replica 3 would provide better protection for split-brains. Thanks, Vijay
----- Original Message -----
From: "Andrew Lau" <andrew@andrewklau.com> To: "users" <users@ovirt.org> Sent: Friday, July 18, 2014 4:50:31 AM Subject: [ovirt-users] Can we debug some truths/myths/facts about hosted-engine and gluster?
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
My current setup is hosted engine, configured w/ gluster storage as described in my blog post, but with three hosts and replica 3 volumes. Only issue I've seen is an errant message about the Hosted Engine being down following an engine migration. The engine does migrate successfully, though. RE your bug, what do you use for a mount point for the nfs storage? Jason
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions.
"
Until the documentation gets updated, I hope this serves as a useful notice at least to save people some of the headaches I hit like hosted-engine starting up multiple VMs because of above issue.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
Thanks, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Jason Brooks" <jbrooks@redhat.com> To: "Andrew Lau" <andrew@andrewklau.com> Cc: "users" <users@ovirt.org> Sent: Tuesday, July 22, 2014 8:29:46 AM Subject: Re: [ovirt-users] Can we debug some truths/myths/facts about hosted-engine and gluster?
----- Original Message -----
From: "Andrew Lau" <andrew@andrewklau.com> To: "users" <users@ovirt.org> Sent: Friday, July 18, 2014 4:50:31 AM Subject: [ovirt-users] Can we debug some truths/myths/facts about hosted-engine and gluster?
Hi all,
As most of you have got hints from previous messages, hosted engine won't work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
My current setup is hosted engine, configured w/ gluster storage as described in my blog post, but with three hosts and replica 3 volumes.
Only issue I've seen is an errant message about the Hosted Engine being down following an engine migration. The engine does migrate successfully, though.
RE your bug, what do you use for a mount point for the nfs storage?
In the log you attached to your bug, it looks like you're using localhost as the nfs mount point. I use a dns name that resolves to the virtual IP hosted by ctdb. So, you're only ever talking to one nfs server at a time, and failover between the nfs hosts is handled by ctdb. Anyway, like I said, my main testing rig is now using this configuration, help me try and break it. :)
Jason
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions.
"
Until the documentation gets updated, I hope this serves as a useful notice at least to save people some of the headaches I hit like hosted-engine starting up multiple VMs because of above issue.
Now my question, does this theory prevent a scenario of perhaps something like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to provide a last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
Thanks, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 23/07/2014 1:45 am, "Jason Brooks" <jbrooks@redhat.com> wrote:
----- Original Message -----
From: "Jason Brooks" <jbrooks@redhat.com> To: "Andrew Lau" <andrew@andrewklau.com> Cc: "users" <users@ovirt.org> Sent: Tuesday, July 22, 2014 8:29:46 AM Subject: Re: [ovirt-users] Can we debug some truths/myths/facts
----- Original Message -----
From: "Andrew Lau" <andrew@andrewklau.com> To: "users" <users@ovirt.org> Sent: Friday, July 18, 2014 4:50:31 AM Subject: [ovirt-users] Can we debug some truths/myths/facts about hosted-engine and gluster?
Hi all,
As most of you have got hints from previous messages, hosted engine
won't
work on gluster . A quote from BZ1097639
"Using hosted engine with Gluster backed storage is currently something we really warn against.
My current setup is hosted engine, configured w/ gluster storage as described in my blog post, but with three hosts and replica 3 volumes.
Only issue I've seen is an errant message about the Hosted Engine being down following an engine migration. The engine does migrate successfully,
about hosted-engine and gluster? though. That was fixed in 3.4.3 I believe, although when it happened to me my engine didn't migrate it just sat there.
RE your bug, what do you use for a mount point for the nfs storage?
In the log you attached to your bug, it looks like you're using localhost as the nfs mount point. I use a dns name that resolves to the virtual IP hosted by ctdb. So, you're only ever talking to one nfs server at a time, and failover between the nfs hosts is handled by ctdb.
I also tried your setup, but hit other complications. I used localhost in an old setup, previously as I was under the assumption when accessing anything gluster related, the connection point only provides the volume info and you connect to any server in the volume group.
Anyway, like I said, my main testing rig is now using this configuration, help me try and break it. :)
rm -rf / Jokes aside, are you able to reboot a server without losing the VM ? My experience with ctdb (based on your blog) was even with the "floating/virtual IP" it wasn't fast enough, or something in the gluster layer delayed the failover. Either way, the VM goes into paused state and can't be resumed.
Jason
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions.
"
Until the documentation gets updated, I hope this serves as a useful notice at least to save people some of the headaches I hit like hosted-engine starting up multiple VMs because of above issue.
Now my question, does this theory prevent a scenario of perhaps
like a gluster replicated volume being mounted as a glusterfs filesystem and then re-exported as the native kernel NFS share for the hosted-engine to consume? It could then be possible to chuck ctdb in there to
something provide a
last resort failover solution. I have tried myself and suggested it to two people who are running a similar setup. Now using the native kernel NFS server for hosted-engine and they haven't reported as many issues. Curious, could anyone validate my theory on this?
Thanks, Andrew
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
RE your bug, what do you use for a mount point for the nfs storage?
In the log you attached to your bug, it looks like you're using localhost as the nfs mount point. I use a dns name that resolves to the virtual IP hosted by ctdb. So, you're only ever talking to one nfs server at a time, and failover between the nfs hosts is handled by ctdb.
I also tried your setup, but hit other complications. I used localhost in an old setup, previously as I was under the assumption when accessing anything gluster related, the connection point only provides the volume info and you connect to any server in the volume group.
As I understand it, with Gluster's nfs, the server you mount is the only one you're accessing directly, which is why you need to use something else to distribute the load, like round robin dns, with gluster's nfs.
Anyway, like I said, my main testing rig is now using this configuration, help me try and break it. :)
rm -rf /
Jokes aside, are you able to reboot a server without losing the VM ? My experience with ctdb (based on your blog) was even with the "floating/virtual IP" it wasn't fast enough, or something in the gluster layer delayed the failover. Either way, the VM goes into paused state and can't be resumed.
I have rebooted my hosts without issue. If I want to reboot the host that's serving the nfs storage, I've stopped ctdb first on that host, to make it hand off the nfs -- I've done this out of caution, but I should try just pulling the plug, too. The main source of VM pausing I've seen is when you have two nodes, one goes down, and the gluster quorum business goes into effect. With my current 3 node, replica 3 setup, gluster stays happy wrt quorum. I'll be sure to post about it if I have problems, but it's been working well for me. Jason
participants (6)
-
Andrew Lau -
Itamar Heim -
Jason Brooks -
Jiri Moskovcak -
Pranith Kumar Karampuri -
Vijay Bellur