[ovirt-users] [Gluster-devel] Can we debug some truths/myths/facts about hosted-engine and gluster?

Sat Jul 19 06:58:12 UTC 2014

On 07/19/2014 11:25 AM, Andrew Lau wrote:
>
>
> On Sat, Jul 19, 2014 at 12:03 AM, Pranith Kumar Karampuri 
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>
>
>     On 07/18/2014 05:43 PM, Andrew Lau wrote:
>>      
>>
>>     On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur
>>     <vbellur at redhat.com <mailto:vbellur at redhat.com>> wrote:
>>
>>         [Adding gluster-devel]
>>
>>
>>         On 07/18/2014 05:20 PM, Andrew Lau wrote:
>>
>>             Hi all,
>>
>>             As most of you have got hints from previous messages,
>>             hosted engine
>>             won't work on gluster . A quote from BZ1097639
>>
>>             "Using hosted engine with Gluster backed storage is
>>             currently something
>>             we really warn against.
>>
>>
>>             I think this bug should be closed or re-targeted at
>>             documentation, because there is nothing we can do here.
>>             Hosted engine assumes that all writes are atomic and
>>             (immediately) available for all hosts in the cluster.
>>             Gluster violates those assumptions.
>>             "
>>
>>         I tried going through BZ1097639 but could not find much
>>         detail with respect to gluster there.
>>
>>         A few questions around the problem:
>>
>>         1. Can somebody please explain in detail the scenario that
>>         causes the problem?
>>
>>         2. Is hosted engine performing synchronous writes to ensure
>>         that writes are durable?
>>
>>         Also, if there is any documentation that details the hosted
>>         engine architecture that would help in enhancing our
>>         understanding of its interactions with gluster.
>>
>>
>>             
>>
>>             Now my question, does this theory prevent a scenario of
>>             perhaps
>>             something like a gluster replicated volume being mounted
>>             as a glusterfs
>>             filesystem and then re-exported as the native kernel NFS
>>             share for the
>>             hosted-engine to consume? It could then be possible to
>>             chuck ctdb in
>>             there to provide a last resort failover solution. I have
>>             tried myself
>>             and suggested it to two people who are running a similar
>>             setup. Now
>>             using the native kernel NFS server for hosted-engine and
>>             they haven't
>>             reported as many issues. Curious, could anyone validate
>>             my theory on this?
>>
>>
>>         If we obtain more details on the use case and obtain gluster
>>         logs from the failed scenarios, we should be able to
>>         understand the problem better. That could be the first step
>>         in validating your theory or evolving further recommendations :).
>>
>>
>>      I'm not sure how useful this is, but Jiri Moskovcak tracked
>>     this down in an off list message.
>>
>>      Message Quote:
>>
>>      ==
>>
>>     We were able to track it down to this (thanks Andrew for
>>     providing the testing setup):
>>
>>     -b686-4363-bb7e-dba99e5789b6/ha_agent service_type=hosted-engine'
>>     Traceback (most recent call last):
>>     File
>>     "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>>     line 165, in handle
>>       response = "success " + self._dispatch(data)
>>     File
>>     "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py",
>>     line 261, in _dispatch
>>       .get_all_stats_for_service_type(**options)
>>     File
>>     "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>>     line 41, in get_all_stats_for_service_type
>>       d = self.get_raw_stats_for_service_type(storage_dir, service_type)
>>     File
>>     "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
>>     line 74, in get_raw_stats_for_service_type
>>       f = os.open(path, direct_flag | os.O_RDONLY)
>>     OSError: [Errno 116] Stale file handle:
>>     '/rhev/data-center/mnt/localhost:_mnt_hosted-engine/c898fd2a-b686-4363-bb7e-dba99e5789b6/ha_agent/hosted-engine.metadata'
>     Andrew/Jiri,
>             Would it be possible to post gluster logs of both the
>     mount and bricks on the bz? I can take a look at it once. If I
>     gather nothing then probably I will ask for your help in
>     re-creating the issue.
>
>     Pranith
>
>
> Unfortunately, I don't have the logs for that setup any more.. I'll 
> try replicate when I get a chance. If I understand the comment from 
> the BZ, I don't think it's a gluster bug per-say, more just how 
> gluster does its replication.
hi Andrew,
          Thanks for that. I couldn't come to any conclusions because no 
logs were available. It is unlikely that self-heal is involved because 
there were no bricks going down/up according to the bug description.

Pranith
>
>
>>
>>     It's definitely connected to the storage which leads us to the
>>     gluster, I'm not very familiar with the gluster so I need to
>>     check this with our gluster gurus.
>>
>>     == 
>>
>>         Thanks,
>>         Vijay
>>
>>
>>
>>
>>     _______________________________________________
>>     Gluster-devel mailing list
>>     Gluster-devel at gluster.org  <mailto:Gluster-devel at gluster.org>
>>     http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140719/102e56bd/attachment-0001.html>