slow performance with export storage on glusterfs
This is a cryptographically signed message in MIME format. --------------ms020509020901010502070100 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi, I am trying realize why is exporting of vm to export storage on glusterfs such slow. I am using oVirt and RHV, both instalations on version 4.1.7. Hosts have dedicated nics for rhevm network - 1gbps, data storage itself is on FC. GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks but I can achieve about 200-400mbit throughput in other applications (we are using it for "cold" data, backups mostly). I am using this glusterfs cluster as backend for export storage. When I am exporting vm I can see only about 60-80mbit throughput. What could be the bottleneck here? Could it be qemu-img utility? vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426= -8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d= 6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4e= a2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f= -d6cf-48cc-ad0c-4a7d979c0c1e Any idea how to make it work faster or what throughput should I expected?= Cheers, Jiri --------------ms020509020901010502070100 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+ 0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101 S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/ VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E 8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf /SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMDE1MjAzM1owLwYJ KoZIhvcNAQkEMSIEILmfZ4a/vSZv57MxXgCAPwjxh9wMgWdW8v78HTqqQ3zIMGwGCSqGSIb3 DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAGii FiUp6Lb3R5JRu32BeDHrAGBL8Ym2pCdXh8zv4dYs3Q3dd/SQHkif4Z3qysUHfPAEQ2gz7N4C W7ALZjblD0RZRy5JlrzGLfCmruPBKmU/cLIAYZk2v78+aEjgi8oiZHSLnhIwCRQheQtxK7rG O8qkWpqZ2jmCstaBlj8kbpVb4WvNC7EbtEJLjXTP4rKAET2daeZdLqPaLx6eYND3l10GymKV k4KLh0o638Eh3fmBLIncOh79nuURZl5Q6K7bflEQacQEzWDw9+9fGf5vWc8FzTzrn6w9RPwZ NCcAVdR8SAU274u0cFWPnj+UywsJRKywJ31UWbQGU8ONy7MsAoAAAAAAAAA= --------------ms020509020901010502070100--
Hi Jiri, Sorry for the delay. Do you experience the same issue for non-gluster domains? In order to profile your gluster volume while export is in progress, follow the instructions in this link [1]. (Please execute "gluster volume profile <volname> start " and then "gluster volume profile <volname> info") [1] https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.1/html/Admin... *Regards,* *Shani Leviim* On Mon, Nov 20, 2017 at 5:20 PM, Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks but I can achieve about 200-400mbit throughput in other applications (we are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2-b081-7c6288fbb636/images/ ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
Cheers,
Jiri
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks but I can achieve about 200-400mbit throughput in other applications (we are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06 /usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41: _rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
gluster storage operations are using fuse mount - so every write: - travel to the kernel - travel back to the gluster fuse helper process - travel to all 3 replicas - replication is done on client side - return to kernel when all writes succeeded - return to caller So gluster will never set any speed record. Additionally, you are copying from raw lv on FC - qemu-img cannot do anything smart and avoid copying unused clusters. Instead if copies gigabytes of zeros from FC. However 7.5-10 MiB/s sounds too slow. I would try to test with dd - how much time it takes to copy the same image from FC to your gluster storage? dd if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__ bs=8M oflag=direct status=progress If dd can do this faster, please ask on qemu-discuss mailing list: https://lists.nongnu.org/mailman/listinfo/qemu-discuss If both give similar results, I think asking in gluster mailing list about this can help. Maybe your gluster setup can be optimized. Nir
Cheers,
Jiri
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a cryptographically signed message in MIME format. --------------ms040706070909040905040207 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hi, On 11/22/2017 07:30 PM, Nir Soffer wrote:
On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <jiri.s= lezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote: =20 Hi, =20 I am trying realize why is exporting of vm to export storage on glusterfs such slow. =20 I am using oVirt and RHV, both instalations on version 4.1.7. =20 Hosts have dedicated nics for rhevm network - 1gbps, data storage i= tself is on FC. =20 GlusterFS cluster lives separate on 4 dedicated hosts. It has slow = disks but I can achieve about 200-400mbit throughput in other application= s (we are using it for "cold" data, backups mostly). =20 I am using this glusterfs cluster as backend for export storage. Wh= en I am exporting vm I can see only about 60-80mbit throughput. =20 What could be the bottleneck here? =20 Could it be qemu-img utility? =20 vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0 0.0 354212 29148 ?=C2= =A0 =C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A00:06 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-100= 5-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57a= cd5f-d6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a= 392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c5= 7acd5f-d6cf-48cc-ad0c-4a7d979c0c1e =20 Any idea how to make it work faster or what throughput should I expected? =20 =20 gluster storage operations are using fuse mount - so every write: - travel to the kernel - travel back to the gluster fuse helper process - travel to all 3 replicas - replication is done on client side - return to kernel when all writes succeeded - return to caller =20 So gluster will never set any speed record. =20 Additionally, you are copying from raw lv on FC - qemu-img cannot do anything smart and avoid copying unused clusters. Instead if copies gigabytes of=
zeros from FC.
ok, it does make sense
However 7.5-10 MiB/s sounds too slow. =20 I would try to test with dd - how much time it takes to copy the same image from FC to your gluster storage? =20 dd if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-10= 05-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57= acd5f-d6cf-48cc-ad0c-4a7d979c0c1e of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-= a392-4ea2-b081-7c6288fbb636/__test__ bs=3D8M oflag=3Ddirect status=3Dprogress
unfrotunately dd performs the same 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
If dd can do this faster, please ask on qemu-discuss mailing list: https://lists.nongnu.org/mailman/listinfo/qemu-discuss =20 If both give similar results, I think asking in gluster mailing list about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance. I will investigate the gluster side and also will try Export on NFS share= =2E Cheers, Jiri
=20 Nir =C2=A0 =20 =20 Cheers, =20 Jiri =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users =20
--------------ms040706070909040905040207 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+ 0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101 S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/ VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E 8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf /SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMzExMjY0M1owLwYJ KoZIhvcNAQkEMSIEIHDYuAnjwPWpgh4XDFXPeS0Y1CjWUEbKw7qRlk6OR0hFMGwGCSqGSIb3 DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBADEU mf5wA7QyZ+c5nhjrS/RrBsua5NDZ+vk9/0bOUu17nJx17wOHKcFuqjTZV1kptc1LB0GrZpsE GBBbtSYYlpTER54p3EDaV9LowE570S+kEUGyErANpyJmUIdJx1w9cPlIGBK9Kw23scDgnfy/ W21EWJyjlsIauzQ61tRl5NyZS9K1zOytQciB1lm/ac8XIaaHRq5VXolkLYWBW5HmBZoNifGv M059/+cH+kzYD3zb/Py6/hbYbZEawz0f5W2dAMIxU1DgX7aJGJ2gz83WfCZBI1j7GE58y9Ww DGcrZ9ALCch3Il+uGKCDvmb8kSqIqaT1rCbILkA2nvSAIC2qSjcAAAAAAAA= --------------ms040706070909040905040207--
This is a cryptographically signed message in MIME format. --------------ms060403050704060409050400 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable well, another idea when I did not use the direct flag, the performace was much better 15787360256 bytes (16 GB) copied, 422.955159 s, 37.3 MB/s probably qemu-img uses direct write too and I understand why. But in case of backup it is not as hot I think. Is there a chance to modify this behavior for backup case? Is it a good idea? Should I fill RFE? Cheers, Jiri On 11/23/2017 12:26 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka wrote:
Hi, =20 On 11/22/2017 07:30 PM, Nir Soffer wrote:
On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <jiri.= slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage = itself is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow= disks but I can achieve about 200-400mbit throughput in other applicatio= ns (we are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. W= hen I am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0 0.0 354212 29148 ?=C2= =A0 =C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A00:06 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-10= 05-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57= acd5f-d6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-= a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c= 57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
gluster storage operations are using fuse mount - so every write: - travel to the kernel - travel back to the gluster fuse helper process - travel to all 3 replicas - replication is done on client side - return to kernel when all writes succeeded - return to caller
So gluster will never set any speed record.
Additionally, you are copying from raw lv on FC - qemu-img cannot do anything smart and avoid copying unused clusters. Instead if copies gigabytes o= f zeros from FC. =20 ok, it does make sense =20 However 7.5-10 MiB/s sounds too slow.
I would try to test with dd - how much time it takes to copy the same image from FC to your gluster storage?
dd if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1= 005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c5= 7acd5f-d6cf-48cc-ad0c-4a7d979c0c1e of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499= -a392-4ea2-b081-7c6288fbb636/__test__ bs=3D8M oflag=3Ddirect status=3Dprogress =20 unfrotunately dd performs the same =20 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s =20 =20 If dd can do this faster, please ask on qemu-discuss mailing list: https://lists.nongnu.org/mailman/listinfo/qemu-discuss
If both give similar results, I think asking in gluster mailing list about this can help. Maybe your gluster setup can be optimized. =20 ok, this is definitly on the gluster side. Thanks for your guidance. =20 I will investigate the gluster side and also will try Export on NFS sha= re. =20 Cheers, =20 Jiri =20 =20
Nir =C2=A0
Cheers,
Jiri
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
=20 =20 =20 =20 _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users =20
--------------ms060403050704060409050400 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+ 0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101 S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/ VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E 8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf /SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMzExNDM0NlowLwYJ KoZIhvcNAQkEMSIEIMM2KSwC5xIOAF72J62BEBxPnPElG/qlnUVvTlWIGvR0MGwGCSqGSIb3 DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAK1V y3x22QWoL0uYr5QPl+vYfp/UYzk7ngzeGaFUqIL8+SHBWDNuPINgjcHPo4Y5hxjnJ9i9sEVJ qJ/dd/9/PorxVOaZh+ejg2lPw/3fCa7hJmCAt3U6yb1OdkqEVGSAfrCnJHXOlMMBbX/RNODk ooTOGmItuhc8rzdSI5uOTOC/vpXdI4H8nbkzEUNpEADZO9a7wjjKTUezxhQ7TjE2SEBtTqay gqP49+bTZb74O5BqZ352Y5QF7I1AAAV8U3RfIX7mtDlIr6Qhpyv4BSD5uo0vm45lOdoOhjvq R7jshe+gfoyUTDCqeQ+ucQZRBvaAVg1L4IQQmBExo4ezhAnBMHAAAAAAAAA= --------------ms060403050704060409050400--
On Thu, Nov 23, 2017 at 1:43 PM, Jiří Sléžka <jiri.slezka@slu.cz> wrote:
well, another idea
when I did not use the direct flag, the performace was much better
15787360256 bytes (16 GB) copied, 422.955159 s, 37.3 MB/s
That means you were hitting the cache.
probably qemu-img uses direct write too and I understand why. But in case of backup it is not as hot I think. Is there a chance to modify this behavior for backup case? Is it a good idea? Should I fill RFE?
Probably not. We really prefer direct IO to ensure data is consistent. Y.
Cheers,
Jiri
On 11/23/2017 12:26 PM, Jiří Sléžka wrote:
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote:
On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks but I can achieve about 200-400mbit throughput in other applications (we are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2-b081-7c6288fbb636/images/ ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
gluster storage operations are using fuse mount - so every write: - travel to the kernel - travel back to the gluster fuse helper process - travel to all 3 replicas - replication is done on client side - return to kernel when all writes succeeded - return to caller
So gluster will never set any speed record.
Additionally, you are copying from raw lv on FC - qemu-img cannot do anything smart and avoid copying unused clusters. Instead if copies gigabytes of zeros from FC.
ok, it does make sense
However 7.5-10 MiB/s sounds too slow.
I would try to test with dd - how much time it takes to copy the same image from FC to your gluster storage?
dd if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2-b081-7c6288fbb636/__test__ bs=8M oflag=direct status=progress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
If dd can do this faster, please ask on qemu-discuss mailing list: https://lists.nongnu.org/mailman/listinfo/qemu-discuss
If both give similar results, I think asking in gluster mailing list about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share.
Cheers,
Jiri
Nir
Cheers,
Jiri
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Thu, Nov 23, 2017 at 3:33 PM Yaniv Kaul <ykaul@redhat.com> wrote:
On Thu, Nov 23, 2017 at 1:43 PM, Jiří Sléžka <jiri.slezka@slu.cz> wrote:
well, another idea
when I did not use the direct flag, the performace was much better
15787360256 bytes (16 GB) copied, 422.955159 s, 37.3 MB/s
That means you were hitting the cache.
probably qemu-img uses direct write too and I understand why. But in case of backup it is not as hot I think. Is there a chance to modify this behavior for backup case? Is it a good idea? Should I fill RFE?
Probably not. We really prefer direct IO to ensure data is consistent. Y
I did some research in prehistoric oVirt source, and found why we started to use direct I/O. We have two issues: 1. reading stale data from storage 2. trashing host cache If you don't use direct I/O when accessing shared storage, you risk reading stale data from the kernel buffer cache. This cache may be stale since the kernel does not know anything about other hosts writing to the same storage after the last read from this storage. The -t none option in vdsm was introduced because of https://bugzilla.redhat.com/699976. The qemu bug https://bugzilla.redhat.com/713743 explains the issue: qemu-img was writing disk images using writeback and fillingup the cache buffers which are then flushed by the kernel preventing other processes from accessing the storage. This is particularly bad in cluster environments where time-based algorithms might be in place and accessing the storage within certain timeouts is critical I'm not sure it this issue relevant now. We use now sanlock instead of safelease, (except for export domain still using safelease), and qemu or kernel may have better options to avoid trashing the host cache, or guarantee reliable access to storage. Daivd, do you know if sanlock is effected by trashing the host cache? Adding also qemu-block mailing list. Nir
Cheers,
Jiri
On 11/23/2017 12:26 PM, Jiří Sléžka wrote:
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote:
On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks but I can achieve about 200-400mbit throughput in other applications (we are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06 /usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
-O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
gluster storage operations are using fuse mount - so every write: - travel to the kernel - travel back to the gluster fuse helper process - travel to all 3 replicas - replication is done on client side - return to kernel when all writes succeeded - return to caller
So gluster will never set any speed record.
Additionally, you are copying from raw lv on FC - qemu-img cannot do anything smart and avoid copying unused clusters. Instead if copies gigabytes of zeros from FC.
ok, it does make sense
However 7.5-10 MiB/s sounds too slow.
I would try to test with dd - how much time it takes to copy the same image from FC to your gluster storage?
dd
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
of=/rhev/data-center/mnt/glusterSD/10.20.30.41: _rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__ bs=8M oflag=direct status=progress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
If dd can do this faster, please ask on qemu-discuss mailing list: https://lists.nongnu.org/mailman/listinfo/qemu-discuss
If both give similar results, I think asking in gluster mailing list about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share.
Cheers,
Jiri
Nir
Cheers,
Jiri
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Am 07.12.2017 um 23:45 hat Nir Soffer geschrieben:
The qemu bug https://bugzilla.redhat.com/713743 explains the issue: qemu-img was writing disk images using writeback and fillingup the cache buffers which are then flushed by the kernel preventing other processes from accessing the storage. This is particularly bad in cluster environments where time-based algorithms might be in place and accessing the storage within certain timeouts is critical
I'm not sure it this issue relevant now. We use now sanlock instead of safelease, (except for export domain still using safelease), and qemu or kernel may have better options to avoid trashing the host cache, or guarantee reliable access to storage.
Non-direct means that the data goes through the kernel page cache, and the kernel doesn't know that it won't be needed again, so it will fill up the cache with the image. I'm also not aware that cache coherency is now provided by all backends for shared storage, so O_DIRECT still seems to be the only way to avoid using stale caches. Since the problem is about stale caches, I don't see how the locking mechanism could make a difference. The only thing I can suggest, given that there is a "glusterfs" in the subject line of the email, is that the native gluster driver in QEMU takes a completely different path and never uses the kernel page cache, which should make both problems disappear. Maybe it would be worth having a look at this. Kevin
On Fri, Dec 8, 2017 at 4:18 PM Kevin Wolf <kwolf@redhat.com> wrote:
Am 07.12.2017 um 23:45 hat Nir Soffer geschrieben:
The qemu bug https://bugzilla.redhat.com/713743 explains the issue: qemu-img was writing disk images using writeback and fillingup the cache buffers which are then flushed by the kernel preventing other processes from accessing the storage. This is particularly bad in cluster environments where time-based algorithms might be in place and accessing the storage within certain timeouts is critical
I'm not sure it this issue relevant now. We use now sanlock instead of safelease, (except for export domain still using safelease), and qemu or kernel may have better options to avoid trashing the host cache, or guarantee reliable access to storage.
Non-direct means that the data goes through the kernel page cache, and the kernel doesn't know that it won't be needed again, so it will fill up the cache with the image.
I'm also not aware that cache coherency is now provided by all backends for shared storage, so O_DIRECT still seems to be the only way to avoid using stale caches. Since the problem is about stale caches, I don't see how the locking mechanism could make a difference.
The only thing I can suggest, given that there is a "glusterfs" in the subject line of the email, is that the native gluster driver in QEMU takes a completely different path and never uses the kernel page cache, which should make both problems disappear. Maybe it would be worth having a look at this.
This is an interesting direction - currently oVirt does not use native glusterfs for qemu-img operation, only for running vms. We still use the fuse based glusterfs mount for storage operations like copying and converting images.
Kevin
On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka@slu.cz> wrote:
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote:
On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks but I can achieve about 200-400mbit throughput in other applications (we are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06 /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e -O raw /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2-b081-7c6288fbb636/images/ ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
gluster storage operations are using fuse mount - so every write: - travel to the kernel - travel back to the gluster fuse helper process - travel to all 3 replicas - replication is done on client side - return to kernel when all writes succeeded - return to caller
So gluster will never set any speed record.
Additionally, you are copying from raw lv on FC - qemu-img cannot do anything smart and avoid copying unused clusters. Instead if copies gigabytes of zeros from FC.
ok, it does make sense
However 7.5-10 MiB/s sounds too slow.
I would try to test with dd - how much time it takes to copy the same image from FC to your gluster storage?
dd if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ export/81094499-a392-4ea2-b081-7c6288fbb636/__test__ bs=8M oflag=direct status=progress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
If dd can do this faster, please ask on qemu-discuss mailing list: https://lists.nongnu.org/mailman/listinfo/qemu-discuss
If both give similar results, I think asking in gluster mailing list about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share.
[Adding gluster users ml] Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shani on how to run this) while performing the dd operation above.
Cheers,
Jiri
Nir
Cheers,
Jiri
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This is a cryptographically signed message in MIME format. --------------ms000801020502000000010003 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 11/24/2017 06:41 AM, Sahina Bose wrote:
=20 =20 On Thu, Nov 23, 2017 at 4:56 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <jiri.= slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote: =20 Hi, =20 On 11/22/2017 07:30 PM, Nir Soffer wrote: > On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <= jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > >=C2=A0 =C2=A0 =C2=A0Hi, > >=C2=A0 =C2=A0 =C2=A0I am trying realize why is exporting of vm to = export storage on >=C2=A0 =C2=A0 =C2=A0glusterfs such slow. > >=C2=A0 =C2=A0 =C2=A0I am using oVirt and RHV, both instalations on= version 4.1.7. > >=C2=A0 =C2=A0 =C2=A0Hosts have dedicated nics for rhevm network - = 1gbps, data storage itself >=C2=A0 =C2=A0 =C2=A0is on FC. > >=C2=A0 =C2=A0 =C2=A0GlusterFS cluster lives separate on 4 dedicate= d hosts. It has slow disks >=C2=A0 =C2=A0 =C2=A0but I can achieve about 200-400mbit throughput= in other applications (we >=C2=A0 =C2=A0 =C2=A0are using it for "cold" data, backups mostly).=
> >=C2=A0 =C2=A0 =C2=A0I am using this glusterfs cluster as backend f=
or export
storage. When I >=C2=A0 =C2=A0 =C2=A0am exporting vm I can see only about 60-80mbit=
throughput.
> >=C2=A0 =C2=A0 =C2=A0What could be the bottleneck here? > >=C2=A0 =C2=A0 =C2=A0Could it be qemu-img utility? > >=C2=A0 =C2=A0 =C2=A0vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0=
0.0 354212 29148 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A0= 0:06
>=C2=A0 =C2=A0 =C2=A0/usr/bin/qemu-img convert -p -t none -T none -=
f raw
>=C2=A0 =C2=A0 =C2=A0/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd5=
6a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e2= 7/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0-O raw >=C2=A0 =C2=A0 =C2=A0/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/8109=
4499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1= e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >=C2=A0 =C2=A0 =C2=A0Any idea how to make it work faster or what th=
roughput should I
>=C2=A0 =C2=A0 =C2=A0expected? > > > gluster storage operations are using fuse mount - so every write:=
> - travel to the kernel > - travel back to the gluster fuse helper process > - travel to all 3 replicas - replication is done on client side > - return to kernel when all writes succeeded > - return to caller > > So gluster will never set any speed record. > > Additionally, you are copying from raw lv on FC - qemu-img cannot=
do
> anything > smart and avoid copying unused clusters. Instead if copies gigabytes of > zeros > from FC. =20 ok, it does make sense =20 > However 7.5-10 MiB/s sounds too slow. > > I would try to test with dd - how much time it takes to copy > the same image from FC to your gluster storage? > > dd > if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd=
56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e= 27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/810=
94499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=3D8M oflag=3Ddirect status=3Dprogress =20 unfrotunately dd performs the same =20 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s =20 =20 > If dd can do this faster, please ask on qemu-discuss mailing list=
:
> https://lists.nongnu.org/mailman/listinfo/qemu-discuss <https://lists.nongnu.org/mailman/listinfo/qemu-discuss> > > If both give similar results, I think asking in gluster mailing l=
ist
> about this can help. Maybe your gluster setup can be optimized. =20 ok, this is definitly on the gluster side. Thanks for your guidance=
=2E
=20 I will investigate the gluster side and also will try Export on NFS=
share. =20 =20 [Adding gluster users ml] =20 Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shan=
i
on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8 as mentioned in other posts. Gluster cluster uses really slow (green) disks but without direct io it can achieve throughput around 400mbit/s. This storage is used mostly for backup purposes. It is not used as a vm storage. In my case it would be nice not to use direct io in export case but I understand why it might not be wise. Cheers, Jiri
=20 =C2=A0 =20 =20 Cheers, =20 Jiri =20 =20 > > Nir > =C2=A0 > > >=C2=A0 =C2=A0 =C2=A0Cheers, > >=C2=A0 =C2=A0 =C2=A0Jiri > > >=C2=A0 =C2=A0 =C2=A0______________________________________________= _ >=C2=A0 =C2=A0 =C2=A0Users mailing list >=C2=A0 =C2=A0 =C2=A0Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> > =20 =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> =20 =20
--------------ms000801020502000000010003 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+ 0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101 S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/ VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E 8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf /SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyNDA4NDgyOVowLwYJ KoZIhvcNAQkEMSIEICc3XTVcQ/q6/38okZAzf1u/2L2Kvnr7lC0WPziYo539MGwGCSqGSIb3 DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBADBi 5DSVYHZZJJKx30+0/xl2OX/7btvMHwfEOD4uIYjTxA2PXjZztLBcyXV7+vPX/SJzGTz38SH5 /FvzDOVXAAaaF0X9Yf+LHLUT1yEY/oc8Mabjbwkg8GzIxUkuTeSoidGnaGaCO26X2sD+Txq/ HZYi0oVqrSVN045GHYi+sBDh7t6QnOB9D/ckidlajCl2iZ2bOe/q19aisAJcP1VrH5H0Mtsc YHS/SlwN32PwO6kS/JZuKVjn8PPiVh6yKeNiESxcMAVbhybq+fGAvn9UZcpcAOzs/8qleVLK 3RT33LCh1QH1COGuI/KeXRCjN/LRBzJ/2w2BDOOg5/zFggSoaFEAAAAAAAA= --------------ms000801020502000000010003--
What about mounting over nfs instead of the fuse client. Or maybe libgfapi. Is that available for export domains On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka@slu.cz> wrote:
On 11/24/2017 06:41 AM, Sahina Bose wrote:
On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote: > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz
<mailto:jiri.slezka@slu.cz>
> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hi, > > I am trying realize why is exporting of vm to export storage on > glusterfs such slow. > > I am using oVirt and RHV, both instalations on version 4.1.7. > > Hosts have dedicated nics for rhevm network - 1gbps, data storage itself > is on FC. > > GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks > but I can achieve about 200-400mbit throughput in other applications (we > are using it for "cold" data, backups mostly). > > I am using this glusterfs cluster as backend for export storage. When I > am exporting vm I can see only about 60-80mbit throughput. > > What could be the bottleneck here? > > Could it be qemu-img utility? > > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43
0:06
> /usr/bin/qemu-img convert -p -t none -T none -f raw >
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> -O raw > /rhev/data-center/mnt/glusterSD/10.20.30.41:
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > Any idea how to make it work faster or what throughput should I > expected? > > > gluster storage operations are using fuse mount - so every write: > - travel to the kernel > - travel back to the gluster fuse helper process > - travel to all 3 replicas - replication is done on client side > - return to kernel when all writes succeeded > - return to caller > > So gluster will never set any speed record. > > Additionally, you are copying from raw lv on FC - qemu-img cannot
do
> anything > smart and avoid copying unused clusters. Instead if copies gigabytes of > zeros > from FC.
ok, it does make sense
> However 7.5-10 MiB/s sounds too slow. > > I would try to test with dd - how much time it takes to copy > the same image from FC to your gluster storage? > > dd >
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> of=/rhev/data-center/mnt/glusterSD/10.20.30.41:
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=8M oflag=direct status=progress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
> If dd can do this faster, please ask on qemu-discuss mailing list: > https://lists.nongnu.org/mailman/listinfo/qemu-discuss <https://lists.nongnu.org/mailman/listinfo/qemu-discuss> > > If both give similar results, I think asking in gluster mailing
list
> about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share.
[Adding gluster users ml]
Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shani on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8
as mentioned in other posts. Gluster cluster uses really slow (green) disks but without direct io it can achieve throughput around 400mbit/s.
This storage is used mostly for backup purposes. It is not used as a vm storage.
In my case it would be nice not to use direct io in export case but I understand why it might not be wise.
Cheers,
Jiri
Cheers,
Jiri
> > Nir > > > > Cheers, > > Jiri > > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> > http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> >
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hello If you use Gluster as FUSE mount it's always slower than you expect it to be. If you want to get better performance out of your oVirt/Gluster storage, try the following: - create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks (Virtual disks are located on your Gluster storage volume). - Boot/configure the VM, then use LVM to create VG/LV with 4 stripes (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS. - then install NFS server and export LV you created in previous step, use the NFS export as export domain in oVirt/RHEV. You should get wire speed when you use multiple stripes on Gluster storage, FUSE mount on oVirt host will fan out requests to all 4 servers. Gluster is very good at distributed/parallel workloads, but when you use direct Gluster FUSE mount for Export domain you only have one data stream, which is fragmented even more my multiple writes/reads that Gluster needs to do to save your data on all member servers. On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny@fortnebula.com> wrote:
What about mounting over nfs instead of the fuse client. Or maybe libgfapi. Is that available for export domains
On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka@slu.cz> wrote:
On 11/24/2017 06:41 AM, Sahina Bose wrote:
On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote: > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka@slu.cz
<mailto:jiri.slezka@slu.cz>
> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > > Hi, > > I am trying realize why is exporting of vm to export storage
on
> glusterfs such slow. > > I am using oVirt and RHV, both instalations on version 4.1.7. > > Hosts have dedicated nics for rhevm network - 1gbps, data storage itself > is on FC. > > GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks > but I can achieve about 200-400mbit throughput in other applications (we > are using it for "cold" data, backups mostly). > > I am using this glusterfs cluster as backend for export storage. When I > am exporting vm I can see only about 60-80mbit throughput. > > What could be the bottleneck here? > > Could it be qemu-img utility? > > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43
0:06
> /usr/bin/qemu-img convert -p -t none -T none -f raw > /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> -O raw > /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/images/ ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > Any idea how to make it work faster or what throughput should
I
> expected? > > > gluster storage operations are using fuse mount - so every write: > - travel to the kernel > - travel back to the gluster fuse helper process > - travel to all 3 replicas - replication is done on client side > - return to kernel when all writes succeeded > - return to caller > > So gluster will never set any speed record. > > Additionally, you are copying from raw lv on FC - qemu-img cannot
do
> anything > smart and avoid copying unused clusters. Instead if copies gigabytes of > zeros > from FC.
ok, it does make sense
> However 7.5-10 MiB/s sounds too slow. > > I would try to test with dd - how much time it takes to copy > the same image from FC to your gluster storage? > > dd > if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=8M oflag=direct status=progress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
> If dd can do this faster, please ask on qemu-discuss mailing list: > https://lists.nongnu.org/mailman/listinfo/qemu-discuss <https://lists.nongnu.org/mailman/listinfo/qemu-discuss> > > If both give similar results, I think asking in gluster mailing
list
> about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share.
[Adding gluster users ml]
Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shani on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8
as mentioned in other posts. Gluster cluster uses really slow (green) disks but without direct io it can achieve throughput around 400mbit/s.
This storage is used mostly for backup purposes. It is not used as a vm storage.
In my case it would be nice not to use direct io in export case but I understand why it might not be wise.
Cheers,
Jiri
Cheers,
Jiri
> > Nir > > > > Cheers, > > Jiri > > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> > http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> >
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users
This is a cryptographically signed message in MIME format. --------------ms050302080405030009040507 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Hello,
=20 If you use Gluster as FUSE mount it's always slower than you expect it to be. If you want to get better performance out of your oVirt/Gluster storage= , try the following:=C2=A0 =20 - create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks (Virtual disks are located on your Gluster storage volume). - Boot/configure the VM, then use LVM to create VG/LV with 4 stripes (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS. - then install NFS server and export LV you created in previous step, use the NFS export as export domain in oVirt/RHEV. =20 You should get wire speed when you use multiple stripes on Gluster storage, FUSE mount on oVirt host will fan out requests to all 4 server= s. Gluster is very good at distributed/parallel workloads, but when you us= e direct Gluster FUSE mount for Export domain you only have one data stream, which is fragmented even more my multiple writes/reads that Gluster needs to do to save your data on all member servers.
=20 =20 =20 On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny@fortnebula.com <mailto:donny@fortnebula.com>> wrote: =20 What about mounting over nfs instead of the fuse client. Or maybe libgfapi. Is that available for export domains =20 On Fri, Nov 24, 2017 at 3:48 AM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <ji= ri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote: =20 On 11/24/2017 06:41 AM, Sahina Bose wrote: > > > On Thu, Nov 23, 2017 at 4:56 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BE= ka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrot= e: > >=C2=A0 =C2=A0 =C2=A0Hi, > >=C2=A0 =C2=A0 =C2=A0On 11/22/2017 07:30 PM, Nir Soffer wrote: >=C2=A0 =C2=A0 =C2=A0> On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99= =C3=AD Sl=C3=A9=C5=BEka <jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> >=C2=A0 =C2=A0 =C2=A0> <mailto:jiri.slezka@slu.cz <mailto:jiri.= slezka@slu.cz> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> wrote= : >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Hi, >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0I am trying realize w= hy is exporting of vm to export storage on >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0glusterfs such slow. >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0I am using oVirt and = RHV, both instalations on version 4.1.7. >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Hosts have dedicated = nics for rhevm network - 1gbps, data >=C2=A0 =C2=A0 =C2=A0storage itself >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0is on FC. >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0GlusterFS cluster liv= es separate on 4 dedicated hosts. It has >=C2=A0 =C2=A0 =C2=A0slow disks >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0but I can achieve abo= ut 200-400mbit throughput in other >=C2=A0 =C2=A0 =C2=A0applications (we >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0are using it for "col= d" data, backups mostly). >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0I am using this glust= erfs cluster as backend for export >=C2=A0 =C2=A0 =C2=A0storage. When I >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0am exporting vm I can= see only about 60-80mbit throughput. >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0What could be the bot= tleneck here? >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Could it be qemu-img = utility? >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0vdsm=C2=A0 =C2=A0 =C2= =A0 97739=C2=A0 0.3=C2=A0 0.0 354212 29148 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 S= <l=C2=A0 15:43=C2=A0 =C2=A00:06 >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0/usr/bin/qemu-img con= vert -p -t none -T none -f raw >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0=C2=A0/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945= f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-0= 0f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0-O raw >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 >=C2=A0 =C2=A0 =C2=A0=C2=A0/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__e= xport/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d= -00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Any idea how to make = it work faster or what throughput should I >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0expected? >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> gluster storage operations are using fus= e mount - so every write: >=C2=A0 =C2=A0 =C2=A0> - travel to the kernel >=C2=A0 =C2=A0 =C2=A0> - travel back to the gluster fuse helper=
Thanks for explanation, it is an interesting solution. Cheers, Jiri process
>=C2=A0 =C2=A0 =C2=A0> - travel to all 3 replicas - replication=
is done on
client side >=C2=A0 =C2=A0 =C2=A0> - return to kernel when all writes succe=
eded
>=C2=A0 =C2=A0 =C2=A0> - return to caller >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> So gluster will never set any speed reco=
rd.
>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Additionally, you are copying from raw l=
v on FC -
qemu-img cannot do >=C2=A0 =C2=A0 =C2=A0> anything >=C2=A0 =C2=A0 =C2=A0> smart and avoid copying unused clusters.=
Instead if copies
>=C2=A0 =C2=A0 =C2=A0gigabytes of >=C2=A0 =C2=A0 =C2=A0> zeros >=C2=A0 =C2=A0 =C2=A0> from FC. > >=C2=A0 =C2=A0 =C2=A0ok, it does make sense > >=C2=A0 =C2=A0 =C2=A0> However 7.5-10 MiB/s sounds too slow. >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> I would try to test with dd - how much t=
ime it takes to copy
>=C2=A0 =C2=A0 =C2=A0> the same image from FC to your gluster s=
torage?
>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> dd >=C2=A0 =C2=A0 =C2=A0> if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3=
cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b= 1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0> of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/8=
1094499-a392-4ea2-b081-7c6288fbb636/__test__
>=C2=A0 =C2=A0 =C2=A0> bs=3D8M oflag=3Ddirect status=3Dprogress=
> >=C2=A0 =C2=A0 =C2=A0unfrotunately dd performs the same > >=C2=A0 =C2=A0 =C2=A01778384896 bytes (1.8 GB) copied, 198.5652=
65 s, 9.0 MB/s
> > >=C2=A0 =C2=A0 =C2=A0> If dd can do this faster, please ask on =
qemu-discuss
mailing list: >=C2=A0 =C2=A0 =C2=A0> https://lists.nongnu.org/mailman/listinf=
o/qemu-discuss
<https://lists.nongnu.org/mailman/listinfo/qemu-discuss> >=C2=A0 =C2=A0 =C2=A0<https://lists.nongnu.org/mailman/listinfo=
/qemu-discuss
<https://lists.nongnu.org/mailman/listinfo/qemu-discuss>> >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> If both give similar results, I think as=
king in gluster
mailing list >=C2=A0 =C2=A0 =C2=A0> about this can help. Maybe your gluster =
setup can be
optimized. > >=C2=A0 =C2=A0 =C2=A0ok, this is definitly on the gluster side.=
Thanks for your
guidance. > >=C2=A0 =C2=A0 =C2=A0I will investigate the gluster side and al=
so will try
Export on NFS >=C2=A0 =C2=A0 =C2=A0share. > > > [Adding gluster users ml] > > Please provide "gluster volume info" output for the rhv_expor=
t
gluster > volume and also volume profile details (refer to earlier mail=
from Shani > on how to run this) while performing the dd operation above. =20 you can find all this output on https://pastebin.com/sBK01VS8 =20 as mentioned in other posts. Gluster cluster uses really slow (green) disks but without direct io it can achieve throughput around 400mbit/s. =20 This storage is used mostly for backup purposes. It is not used=
as a vm storage. =20 In my case it would be nice not to use direct io in export case=
but I understand why it might not be wise. =20 Cheers, =20 Jiri =20 > > =C2=A0 > > >=C2=A0 =C2=A0 =C2=A0Cheers, > >=C2=A0 =C2=A0 =C2=A0Jiri > > >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Nir >=C2=A0 =C2=A0 =C2=A0> =C2=A0 >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Cheers, >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Jiri >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0_____________________=
__________________________
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Users mailing list >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Users@ovirt.org <mail=
to:Users@ovirt.org>
<mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >=C2=A0 =C2=A0 =C2=A0<mailto:Users@ovirt.org <mailto:Users@ovir=
t.org>
<mailto:Users@ovirt.org <mailto:Users@ovirt.org>>> >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.or=
g/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users> >=C2=A0 =C2=A0 =C2=A0<http://lists.ovirt.org/mailman/listinfo/u=
sers
<http://lists.ovirt.org/mailman/listinfo/users>> >=C2=A0 =C2=A0 =C2=A0> > > > >=C2=A0 =C2=A0 =C2=A0__________________________________________=
_____
>=C2=A0 =C2=A0 =C2=A0Users mailing list >=C2=A0 =C2=A0 =C2=A0Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.org/mailman/listinfo/us=
ers
<http://lists.ovirt.org/mailman/listinfo/users> >=C2=A0 =C2=A0 =C2=A0<http://lists.ovirt.org/mailman/listinfo/u=
sers
<http://lists.ovirt.org/mailman/listinfo/users>> > > =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 _______________________________________________ Gluster-users mailing list Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> http://lists.gluster.org/mailman/listinfo/gluster-users <http://lists.gluster.org/mailman/listinfo/gluster-users> =20 =20
--------------ms050302080405030009040507 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+ 0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101 S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/ VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E 8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf /SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyOTA5MDgyMVowLwYJ KoZIhvcNAQkEMSIEIFuS5jd9xR3ci09Hj2Qdoq+KmgeiPc6OokdRtOsinA80MGwGCSqGSIb3 DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAIy4 4CxcJOH1NRuorhCJ7I5HuEeRF31ZM86zderH2J3RnhpAV3uiThbZ+GVwY7To120L87dXElX/ /cdN0apblltVvhIcjASh61dWCgGGSPxGSyfv2P30E3oHsyf8LGMhRinfVbJxbUpy8pUjDwJM 40DkmrSzB1gE9mJKpynqlBrcrZ90eM8DGxZ30Fwh2GikXrd0XYHKcAmNG/NJYHDxosMFyqrK Odr1e117HUL7BIJHVMakjyXPAfisvyiCMxK9CaNMjkdFY/4bAfSEqIphUPxqONYntgI/zgoR DVg5i1dh50f9M/KuX1BuAOUlXotIWgc22XNjRzF/7fbWk0Y/l5IAAAAAAAA= --------------ms050302080405030009040507--
participants (8)
-
Dmitri Chebotarov -
Donny Davis -
Jiří Sléžka -
Kevin Wolf -
Nir Soffer -
Sahina Bose -
Shani Leviim -
Yaniv Kaul