On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hi,

I am trying realize why is exporting of vm to export storage on
glusterfs such slow.

I am using oVirt and RHV, both instalations on version 4.1.7.

Hosts have dedicated nics for rhevm network - 1gbps, data storage itself
is on FC.

GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks
but I can achieve about 200-400mbit throughput in other applications (we
are using it for "cold" data, backups mostly).

I am using this glusterfs cluster as backend for export storage. When I
am exporting vm I can see only about 60-80mbit throughput.

What could be the bottleneck here?

Could it be qemu-img utility?

vdsm      97739  0.3  0.0 354212 29148 ?        S<l  15:43   0:06
/usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
-O raw
/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e

Any idea how to make it work faster or what throughput should I expected?

gluster storage operations are using fuse mount - so every write:
- travel to the kernel
- travel back to the gluster fuse helper process
- travel to all 3 replicas - replication is done on client side
- return to kernel when all writes succeeded
- return to caller

So gluster will never set any speed record.

Additionally, you are copying from raw lv on FC - qemu-img cannot do anything
smart and avoid copying unused clusters. Instead if copies gigabytes of zeros
from FC.

However 7.5-10 MiB/s sounds too slow.

I would try to test with dd - how much time it takes to copy
the same image from FC to your gluster storage?

dd if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__ bs=8M oflag=direct status=progress

If dd can do this faster, please ask on qemu-discuss mailing list:
https://lists.nongnu.org/mailman/listinfo/qemu-discuss

If both give similar results, I think asking in gluster mailing list
about this can help. Maybe your gluster setup can be optimized.

Nir
 

Cheers,

Jiri


_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users