This is a cryptographically signed message in MIME format. --------------ms000801020502000000010003 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 11/24/2017 06:41 AM, Sahina Bose wrote:
=20 =20 On Thu, Nov 23, 2017 at 4:56 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <jiri.= slezka@slu.cz <mailto:jiri.slezka@slu.cz>> wrote: =20 Hi, =20 On 11/22/2017 07:30 PM, Nir Soffer wrote: > On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <= jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: > >=C2=A0 =C2=A0 =C2=A0Hi, > >=C2=A0 =C2=A0 =C2=A0I am trying realize why is exporting of vm to = export storage on >=C2=A0 =C2=A0 =C2=A0glusterfs such slow. > >=C2=A0 =C2=A0 =C2=A0I am using oVirt and RHV, both instalations on= version 4.1.7. > >=C2=A0 =C2=A0 =C2=A0Hosts have dedicated nics for rhevm network - = 1gbps, data storage itself >=C2=A0 =C2=A0 =C2=A0is on FC. > >=C2=A0 =C2=A0 =C2=A0GlusterFS cluster lives separate on 4 dedicate= d hosts. It has slow disks >=C2=A0 =C2=A0 =C2=A0but I can achieve about 200-400mbit throughput= in other applications (we >=C2=A0 =C2=A0 =C2=A0are using it for "cold" data, backups mostly).=
> >=C2=A0 =C2=A0 =C2=A0I am using this glusterfs cluster as backend f=
or export
storage. When I >=C2=A0 =C2=A0 =C2=A0am exporting vm I can see only about 60-80mbit=
throughput.
> >=C2=A0 =C2=A0 =C2=A0What could be the bottleneck here? > >=C2=A0 =C2=A0 =C2=A0Could it be qemu-img utility? > >=C2=A0 =C2=A0 =C2=A0vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0=
0.0 354212 29148 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A0= 0:06
>=C2=A0 =C2=A0 =C2=A0/usr/bin/qemu-img convert -p -t none -T none -=
f raw
>=C2=A0 =C2=A0 =C2=A0/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd5=
6a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e2= 7/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0-O raw >=C2=A0 =C2=A0 =C2=A0/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/8109=
4499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1= e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >=C2=A0 =C2=A0 =C2=A0Any idea how to make it work faster or what th=
roughput should I
>=C2=A0 =C2=A0 =C2=A0expected? > > > gluster storage operations are using fuse mount - so every write:=
> - travel to the kernel > - travel back to the gluster fuse helper process > - travel to all 3 replicas - replication is done on client side > - return to kernel when all writes succeeded > - return to caller > > So gluster will never set any speed record. > > Additionally, you are copying from raw lv on FC - qemu-img cannot=
do
> anything > smart and avoid copying unused clusters. Instead if copies gigabytes of > zeros > from FC. =20 ok, it does make sense =20 > However 7.5-10 MiB/s sounds too slow. > > I would try to test with dd - how much time it takes to copy > the same image from FC to your gluster storage? > > dd > if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd=
56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e= 27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/810=
94499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=3D8M oflag=3Ddirect status=3Dprogress =20 unfrotunately dd performs the same =20 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s =20 =20 > If dd can do this faster, please ask on qemu-discuss mailing list=
:
> https://lists.nongnu.org/mailman/listinfo/qemu-discuss <https://lists.nongnu.org/mailman/listinfo/qemu-discuss> > > If both give similar results, I think asking in gluster mailing l=
ist
> about this can help. Maybe your gluster setup can be optimized. =20 ok, this is definitly on the gluster side. Thanks for your guidance=
=2E
=20 I will investigate the gluster side and also will try Export on NFS=
share. =20 =20 [Adding gluster users ml] =20 Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shan=
i
on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8 as mentioned in other posts. Gluster cluster uses really slow (green) disks but without direct io it can achieve throughput around 400mbit/s. This storage is used mostly for backup purposes. It is not used as a vm storage. In my case it would be nice not to use direct io in export case but I understand why it might not be wise. Cheers, Jiri
=20 =C2=A0 =20 =20 Cheers, =20 Jiri =20 =20 > > Nir > =C2=A0 > > >=C2=A0 =C2=A0 =C2=A0Cheers, > >=C2=A0 =C2=A0 =C2=A0Jiri > > >=C2=A0 =C2=A0 =C2=A0______________________________________________= _ >=C2=A0 =C2=A0 =C2=A0Users mailing list >=C2=A0 =C2=A0 =C2=A0Users@ovirt.org <mailto:Users@ovirt.org> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> > =20 =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users> =20 =20
--------------ms000801020502000000010003 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+ 0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101 S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/ VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E 8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf /SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyNDA4NDgyOVowLwYJ KoZIhvcNAQkEMSIEICc3XTVcQ/q6/38okZAzf1u/2L2Kvnr7lC0WPziYo539MGwGCSqGSIb3 DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBADBi 5DSVYHZZJJKx30+0/xl2OX/7btvMHwfEOD4uIYjTxA2PXjZztLBcyXV7+vPX/SJzGTz38SH5 /FvzDOVXAAaaF0X9Yf+LHLUT1yEY/oc8Mabjbwkg8GzIxUkuTeSoidGnaGaCO26X2sD+Txq/ HZYi0oVqrSVN045GHYi+sBDh7t6QnOB9D/ckidlajCl2iZ2bOe/q19aisAJcP1VrH5H0Mtsc YHS/SlwN32PwO6kS/JZuKVjn8PPiVh6yKeNiESxcMAVbhybq+fGAvn9UZcpcAOzs/8qleVLK 3RT33LCh1QH1COGuI/KeXRCjN/LRBzJ/2w2BDOOg5/zFggSoaFEAAAAAAAA= --------------ms000801020502000000010003--