<div dir="ltr">Hello<div><br></div><div>If you use Gluster as FUSE mount it's always slower than you expect it to be.</div><div>If you want to get better performance out of your oVirt/Gluster storage, try the following: </div><div><br></div><div>- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks (Virtual disks are located on your Gluster storage volume).</div><div>- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.</div><div>- then install NFS server and export LV you created in previous step, use the NFS export as export domain in oVirt/RHEV.</div><div><br></div><div>You should get wire speed when you use multiple stripes on Gluster storage, FUSE mount on oVirt host will fan out requests to all 4 servers.</div><div>Gluster is very good at distributed/parallel workloads, but when you use direct Gluster FUSE mount for Export domain you only have one data stream, which is fragmented even more my multiple writes/reads that Gluster needs to do to save your data on all member servers.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <span dir="ltr"><<a href="mailto:donny@fortnebula.com" target="_blank">donny@fortnebula.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="auto">What about mounting over nfs instead of the fuse client. Or maybe libgfapi. Is that available for export domains</div><br><div class="gmail_quote"><div><div class="h5"><div>On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">On 11/24/2017 06:41 AM, Sahina Bose wrote:<br>
><br>
><br>
> On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a><br>
> <mailto:<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a>>> wrote:<br>
><br>
> Hi,<br>
><br>
> On 11/22/2017 07:30 PM, Nir Soffer wrote:<br>
> > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a> <mailto:<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a>><br>
> > <mailto:<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a> <mailto:<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a>>>> wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> > I am trying realize why is exporting of vm to export storage on<br>
> > glusterfs such slow.<br>
> ><br>
> > I am using oVirt and RHV, both instalations on version 4.1.7.<br>
> ><br>
> > Hosts have dedicated nics for rhevm network - 1gbps, data<br>
> storage itself<br>
> > is on FC.<br>
> ><br>
> > GlusterFS cluster lives separate on 4 dedicated hosts. It has<br>
> slow disks<br>
> > but I can achieve about 200-400mbit throughput in other<br>
> applications (we<br>
> > are using it for "cold" data, backups mostly).<br>
> ><br>
> > I am using this glusterfs cluster as backend for export<br>
> storage. When I<br>
> > am exporting vm I can see only about 60-80mbit throughput.<br>
> ><br>
> > What could be the bottleneck here?<br>
> ><br>
> > Could it be qemu-img utility?<br>
> ><br>
> > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06<br>
> > /usr/bin/qemu-img convert -p -t none -T none -f raw<br>
> > <br>
> /rhev/data-center/2ff6d0ee-<wbr>a10b-473d-b77c-be9149945f5f/<wbr>ff3cd56a-1005-4426-8137-<wbr>8f422c0b47c1/images/ba42cbcc-<wbr>c068-4df8-af3d-00f2077b1e27/<wbr>c57acd5f-d6cf-48cc-ad0c-<wbr>4a7d979c0c1e<br>
> > -O raw<br>
> > <br>
> /rhev/data-center/mnt/<wbr>glusterSD/10.20.30.41:_rhv__<wbr>export/81094499-a392-4ea2-<wbr>b081-7c6288fbb636/images/<wbr>ba42cbcc-c068-4df8-af3d-<wbr>00f2077b1e27/c57acd5f-d6cf-<wbr>48cc-ad0c-4a7d979c0c1e<br>
> ><br>
> > Any idea how to make it work faster or what throughput should I<br>
> > expected?<br>
> ><br>
> ><br>
> > gluster storage operations are using fuse mount - so every write:<br>
> > - travel to the kernel<br>
> > - travel back to the gluster fuse helper process<br>
> > - travel to all 3 replicas - replication is done on client side<br>
> > - return to kernel when all writes succeeded<br>
> > - return to caller<br>
> ><br>
> > So gluster will never set any speed record.<br>
> ><br>
> > Additionally, you are copying from raw lv on FC - qemu-img cannot do<br>
> > anything<br>
> > smart and avoid copying unused clusters. Instead if copies<br>
> gigabytes of<br>
> > zeros<br>
> > from FC.<br>
><br>
> ok, it does make sense<br>
><br>
> > However 7.5-10 MiB/s sounds too slow.<br>
> ><br>
> > I would try to test with dd - how much time it takes to copy<br>
> > the same image from FC to your gluster storage?<br>
> ><br>
> > dd<br>
> > if=/rhev/data-center/2ff6d0ee-<wbr>a10b-473d-b77c-be9149945f5f/<wbr>ff3cd56a-1005-4426-8137-<wbr>8f422c0b47c1/images/ba42cbcc-<wbr>c068-4df8-af3d-00f2077b1e27/<wbr>c57acd5f-d6cf-48cc-ad0c-<wbr>4a7d979c0c1e<br>
> > of=/rhev/data-center/mnt/<wbr>glusterSD/10.20.30.41:_rhv__<wbr>export/81094499-a392-4ea2-<wbr>b081-7c6288fbb636/__test__<br>
> > bs=8M oflag=direct status=progress<br>
><br>
> unfrotunately dd performs the same<br>
><br>
> 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s<br>
><br>
><br>
> > If dd can do this faster, please ask on qemu-discuss mailing list:<br>
> > <a href="https://lists.nongnu.org/mailman/listinfo/qemu-discuss" rel="noreferrer" target="_blank">https://lists.nongnu.org/<wbr>mailman/listinfo/qemu-discuss</a><br>
> <<a href="https://lists.nongnu.org/mailman/listinfo/qemu-discuss" rel="noreferrer" target="_blank">https://lists.nongnu.org/<wbr>mailman/listinfo/qemu-discuss</a>><br>
> ><br>
> > If both give similar results, I think asking in gluster mailing list<br>
> > about this can help. Maybe your gluster setup can be optimized.<br>
><br>
> ok, this is definitly on the gluster side. Thanks for your guidance.<br>
><br>
> I will investigate the gluster side and also will try Export on NFS<br>
> share.<br>
><br>
><br>
> [Adding gluster users ml]<br>
><br>
> Please provide "gluster volume info" output for the rhv_export gluster<br>
> volume and also volume profile details (refer to earlier mail from Shani<br>
> on how to run this) while performing the dd operation above.<br>
<br>
you can find all this output on <a href="https://pastebin.com/sBK01VS8" rel="noreferrer" target="_blank">https://pastebin.com/sBK01VS8</a><br>
<br>
as mentioned in other posts. Gluster cluster uses really slow (green)<br>
disks but without direct io it can achieve throughput around 400mbit/s.<br>
<br>
This storage is used mostly for backup purposes. It is not used as a vm<br>
storage.<br>
<br>
In my case it would be nice not to use direct io in export case but I<br>
understand why it might not be wise.<br>
<br>
Cheers,<br>
<br>
Jiri<br>
<br>
><br>
> <br>
><br>
><br>
> Cheers,<br>
><br>
> Jiri<br>
><br>
><br>
> ><br>
> > Nir<br>
> > <br>
> ><br>
> ><br>
> > Cheers,<br>
> ><br>
> > Jiri<br>
> ><br>
> ><br>
> > _____________________________<wbr>__________________<br>
> > Users mailing list<br>
> > <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>>><br>
> > <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a>><br>
> ><br>
><br>
><br>
><br>
> _____________________________<wbr>__________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a>><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a>><br>
><br>
><br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
</div></div><a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
</blockquote></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>