[ovirt-users] [Gluster-users] slow performance with export storage on glusterfs

Dmitri Chebotarov 4dimach at gmail.com
Tue Nov 28 14:27:51 UTC 2017 

Hello

If you use Gluster as FUSE mount it's always slower than you expect it to
be.
If you want to get better performance out of your oVirt/Gluster storage,
try the following:

- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks
(Virtual disks are located on your Gluster storage volume).
- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
(lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
- then install NFS server and export LV you created in previous step, use
the NFS export as export domain in oVirt/RHEV.

You should get wire speed when you use multiple stripes on Gluster storage,
FUSE mount on oVirt host will fan out requests to all 4 servers.
Gluster is very good at distributed/parallel workloads, but when you use
direct Gluster FUSE mount for Export domain you only have one data stream,
which is fragmented even more my multiple writes/reads that Gluster needs
to do to save your data on all member servers.



On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny at fortnebula.com> wrote:

> What about mounting over nfs instead of the fuse client. Or maybe
> libgfapi. Is that available for export domains
>
> On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka at slu.cz> wrote:
>
>> On 11/24/2017 06:41 AM, Sahina Bose wrote:
>> >
>> >
>> > On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka at slu.cz
>> > <mailto:jiri.slezka at slu.cz>> wrote:
>> >
>> >     Hi,
>> >
>> >     On 11/22/2017 07:30 PM, Nir Soffer wrote:
>> >     > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka at slu.cz
>> <mailto:jiri.slezka at slu.cz>
>> >     > <mailto:jiri.slezka at slu.cz <mailto:jiri.slezka at slu.cz>>> wrote:
>> >     >
>> >     >     Hi,
>> >     >
>> >     >     I am trying realize why is exporting of vm to export storage
>> on
>> >     >     glusterfs such slow.
>> >     >
>> >     >     I am using oVirt and RHV, both instalations on version 4.1.7.
>> >     >
>> >     >     Hosts have dedicated nics for rhevm network - 1gbps, data
>> >     storage itself
>> >     >     is on FC.
>> >     >
>> >     >     GlusterFS cluster lives separate on 4 dedicated hosts. It has
>> >     slow disks
>> >     >     but I can achieve about 200-400mbit throughput in other
>> >     applications (we
>> >     >     are using it for "cold" data, backups mostly).
>> >     >
>> >     >     I am using this glusterfs cluster as backend for export
>> >     storage. When I
>> >     >     am exporting vm I can see only about 60-80mbit throughput.
>> >     >
>> >     >     What could be the bottleneck here?
>> >     >
>> >     >     Could it be qemu-img utility?
>> >     >
>> >     >     vdsm      97739  0.3  0.0 354212 29148 ?        S<l  15:43
>>  0:06
>> >     >     /usr/bin/qemu-img convert -p -t none -T none -f raw
>> >     >
>> >      /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
>> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
>> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> >     >     -O raw
>> >     >
>> >      /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
>> export/81094499-a392-4ea2-b081-7c6288fbb636/images/
>> ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> >     >
>> >     >     Any idea how to make it work faster or what throughput should
>> I
>> >     >     expected?
>> >     >
>> >     >
>> >     > gluster storage operations are using fuse mount - so every write:
>> >     > - travel to the kernel
>> >     > - travel back to the gluster fuse helper process
>> >     > - travel to all 3 replicas - replication is done on client side
>> >     > - return to kernel when all writes succeeded
>> >     > - return to caller
>> >     >
>> >     > So gluster will never set any speed record.
>> >     >
>> >     > Additionally, you are copying from raw lv on FC - qemu-img cannot
>> do
>> >     > anything
>> >     > smart and avoid copying unused clusters. Instead if copies
>> >     gigabytes of
>> >     > zeros
>> >     > from FC.
>> >
>> >     ok, it does make sense
>> >
>> >     > However 7.5-10 MiB/s sounds too slow.
>> >     >
>> >     > I would try to test with dd - how much time it takes to copy
>> >     > the same image from FC to your gluster storage?
>> >     >
>> >     > dd
>> >     > if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
>> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
>> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> >     > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
>> export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
>> >     > bs=8M oflag=direct status=progress
>> >
>> >     unfrotunately dd performs the same
>> >
>> >     1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>> >
>> >
>> >     > If dd can do this faster, please ask on qemu-discuss mailing list:
>> >     > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>> >     <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
>> >     >
>> >     > If both give similar results, I think asking in gluster mailing
>> list
>> >     > about this can help. Maybe your gluster setup can be optimized.
>> >
>> >     ok, this is definitly on the gluster side. Thanks for your guidance.
>> >
>> >     I will investigate the gluster side and also will try Export on NFS
>> >     share.
>> >
>> >
>> > [Adding gluster users ml]
>> >
>> > Please provide "gluster volume info" output for the rhv_export gluster
>> > volume and also volume profile details (refer to earlier mail from Shani
>> > on how to run this) while performing the dd operation above.
>>
>> you can find all this output on https://pastebin.com/sBK01VS8
>>
>> as mentioned in other posts. Gluster cluster uses really slow (green)
>> disks but without direct io it can achieve throughput around 400mbit/s.
>>
>> This storage is used mostly for backup purposes. It is not used as a vm
>> storage.
>>
>> In my case it would be nice not to use direct io in export case but I
>> understand why it might not be wise.
>>
>> Cheers,
>>
>> Jiri
>>
>> >
>> >
>> >
>> >
>> >     Cheers,
>> >
>> >     Jiri
>> >
>> >
>> >     >
>> >     > Nir
>> >     >
>> >     >
>> >     >
>> >     >     Cheers,
>> >     >
>> >     >     Jiri
>> >     >
>> >     >
>> >     >     _______________________________________________
>> >     >     Users mailing list
>> >     >     Users at ovirt.org <mailto:Users at ovirt.org>
>> >     <mailto:Users at ovirt.org <mailto:Users at ovirt.org>>
>> >     >     http://lists.ovirt.org/mailman/listinfo/users
>> >     <http://lists.ovirt.org/mailman/listinfo/users>
>> >     >
>> >
>> >
>> >
>> >     _______________________________________________
>> >     Users mailing list
>> >     Users at ovirt.org <mailto:Users at ovirt.org>
>> >     http://lists.ovirt.org/mailman/listinfo/users
>> >     <http://lists.ovirt.org/mailman/listinfo/users>
>> >
>> >
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20171128/173dac9a/attachment.html>

More information about the Users mailing list