[Users] Gluster storage

[Vijay Bellur]

On 01/17/2014 12:55 PM, Gianluca Cecchi wrote:
> On Fri, Nov 29, 2013 at 11:48 AM, Dan Kenigsberg  wrote:
>> On Fri, Nov 29, 2013 at 04:04:03PM +0530, Vijay Bellur wrote:
>>
>>>
>>> There are two ways in which GlusterFS can be used as a storage domain:
>>>
>>> a) Use gluster native/fuse access with POSIXFS
>>>
>>> b) Use the gluster native storage domain to bypass fuse (with
>>> libgfapi). We are currently addressing an issue in libvirt
>>> (https://bugzilla.redhat.com/show_bug.cgi?id=1017289) to enable
>>> snapshot support with libgfapi. Once this is addressed, we will have
>>> libgfapi support in the native storage domain.
>>
>> It won't be as immediate, since there's a required fix on Vdsm's side
>> (Bug 1022961 - Running a VM from a gluster domain uses mount instead of
>> gluster URI)
>>
>>> Till then, fuse would
>>> be used with native storage domain. You can find more details about
>>> native storage domain here:
>>>
>>> http://www.ovirt.org/Features/GlusterFS_Storage_Domain
> rg/mailman/listinfo/users
>
> Hello,
> revamping here....
> I'm using oVirt 3.3.3 beta1 after upgrade from 3.3.2 on fedora 19
> ovirt beta repo
> It seems bug referred by Vijay (1017289) is still marked as
> "assigned", but actually it is for rhel 6.
> Bug referred by Dan (1022961) is marked as "blocked" but I don't see
> any particular updates since late november. But it is for rhevm, so I
> think it is for rhel 6...
> So what is the situation for fedora 19 and oVirt in upcoming 3.3.3?
> And upcoming fedora 19/20 and 3.4?

>
> I ask because I see that in my qemu command line generated by oVirt there is:
>
> for virtio (virtio-blk) disk
> -drive file=/rhev/data-center/mnt/glusterSD/node01.mydomain:gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/a5e4f67b-50b5-4740-9990-39deb8812445/53408cb0-bcd4-40de-bc69-89d59b7b5bc2,if=none,id=drive-virtio-disk0,format=raw,serial=a5e4f67b-50b5-4740-9990-39deb8812445,cache=none,werror=stop,rerror=stop,aio=threads
>
> for virtio-scsi disk
> -drive file=/rhev/data-center/mnt/glusterSD/node01.mydomain:gvdata/d0b96d4a-62aa-4e9f-b50e-f7a0cb5be291/images/c1477133-6b06-480d-a233-1dae08daf8b3/c2a82c64-9dee-42bb-acf2-65b8081f2edf,if=none,id=drive-scsi0-0-0-0,format=raw,serial=c1477133-6b06-480d-a233-1dae08daf8b3,cache=none,werror=stop,rerror=stop,aio=threads
>
> So it is referred as mount point and not gluster:// way
> Also, the output of "mount" command on hypervisors shows:
>
> node01.mydomain:gvdata on
> /rhev/data-center/mnt/glusterSD/node01.mydomain:gvdata type
> fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
>
> and it seems indeed fuse mount, so not using libgfapi...
> output of
> "lsof -Pp pid" where pid is the qemu process shows libgfapi....:
>
> qemu-syst 2057 qemu  mem       REG              253,0      99440
>          541417 /usr/lib64/libgfapi.so.0.0.0
>
> (btw: strange version 0.0.0 for a release.. not so reassuring ;-)

libgfapi uses a different internal versioning scheme and hence 
compatibility can be handled. However there is work happening in 
glusterfs upstream to change the version scheme for libraries and have 
it compatible with libtool guidelines.

> # ll /usr/lib64/libgfapi.so.0*
> lrwxrwxrwx. 1 root root    17 Jan  7 12:45 /usr/lib64/libgfapi.so.0 ->
> libgfapi.so.0.0.0
> -rwxr-xr-x. 1 root root 99440 Jan  3 13:35 /usr/lib64/libgfapi.so.0.0.0
> )
>
> At page http://www.gluster.org/category/qemu/ there is a schema about
> mount types and benchmarks
>
> 1) FUSE mount
> 2) GlusterFS block driver in QEMU (FUSE bypass)
> 3) Base (VM image accessed directly from brick)
> (
> qemu-system-x86_64 –enable-kvm –nographic -smp 4 -m 2048 -drive
> file=/test/F17,if=virtio,cache=none => /test is brick directory
> )
>
> I have not understood if we are in Base (best performance) or FUSE
> (worst performance)


In this particular blog post, base refers to local disk storage and the 
FUSE performance was measured before a set of optimizations were done in 
GlusterFS. The "Optimize for Virt" action in oVirt ensures that these 
optimizations are enabled. Hence, even with FUSE the performance for VM 
image storage would be somewhere between the worst and best cases.

>
> thanks in advance for clarifications and possible roadmaps...

If you are interested in seeing any new feature related to the 
oVirt-Gluster integration, please feel free to let us know as part of 
the GlusterFS 3.6 planning process [1] which is ongoing now.

- Vijay

[1] http://www.gluster.org/community/documentation/index.php/Planning36