9:20 a.m.
This is a cryptographically signed message in MIME format.
--------------ms020509020901010502070100
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Hi,
I am trying realize why is exporting of vm to export storage on
glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself
is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks
but I can achieve about 200-400mbit throughput in other applications (we
are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I
am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
/usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426=
-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d=
6cf-48cc-ad0c-4a7d979c0c1e
-O raw
/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4e=
a2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f=
-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?=
Cheers,
Jiri
--------------ms020509020901010502070100
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx
MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I
b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd
VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty
qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+
0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim
atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg
oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj
ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB
AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw
AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB
LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr
BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj
egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI
hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF
OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS
WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB
iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101
S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow
ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN
BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw
HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y
ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE
BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT
E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN
MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E
8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX
aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh
tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf
BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V
jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow
DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE
LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f
BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy
c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj
aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw
Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC
AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf
/SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx
VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts
UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd
M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB
MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB
bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl
cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG
SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMDE1MjAzM1owLwYJ
KoZIhvcNAQkEMSIEILmfZ4a/vSZv57MxXgCAPwjxh9wMgWdW8v78HTqqQ3zIMGwGCSqGSIb3
DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG
AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES
MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl
U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC
CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE
BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j
ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAGii
FiUp6Lb3R5JRu32BeDHrAGBL8Ym2pCdXh8zv4dYs3Q3dd/SQHkif4Z3qysUHfPAEQ2gz7N4C
W7ALZjblD0RZRy5JlrzGLfCmruPBKmU/cLIAYZk2v78+aEjgi8oiZHSLnhIwCRQheQtxK7rG
O8qkWpqZ2jmCstaBlj8kbpVb4WvNC7EbtEJLjXTP4rKAET2daeZdLqPaLx6eYND3l10GymKV
k4KLh0o638Eh3fmBLIncOh79nuURZl5Q6K7bflEQacQEzWDw9+9fGf5vWc8FzTzrn6w9RPwZ
NCcAVdR8SAU274u0cFWPnj+UywsJRKywJ31UWbQGU8ONy7MsAoAAAAAAAAA=
--------------ms020509020901010502070100--
7:43 a.m.
Hi Jiri,
Sorry for the delay.
Do you experience the same issue for non-gluster domains?
In order to profile your gluster volume while export is in progress, follow
the instructions in this link [1].
(Please execute "gluster volume profile <volname> start " and then
"gluster
volume profile <volname> info")
[1]
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.1/html/Ad...
*Regards,*
*Shani Leviim*
On Mon, Nov 20, 2017 at 5:20 PM, Jiří Sléžka <jiri.slezka(a)slu.cz> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on
glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself
is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks
but I can achieve about 200-400mbit throughput in other applications (we
are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I
am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
/usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
-O raw
/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/images/
ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
Cheers,
Jiri
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:30 p.m.
On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka(a)slu.cz> wrote:
Hi,
I am trying realize why is exporting of vm to export storage on
glusterfs such slow.
I am using oVirt and RHV, both instalations on version 4.1.7.
Hosts have dedicated nics for rhevm network - 1gbps, data storage itself
is on FC.
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks
but I can achieve about 200-400mbit throughput in other applications (we
are using it for "cold" data, backups mostly).
I am using this glusterfs cluster as backend for export storage. When I
am exporting vm I can see only about 60-80mbit throughput.
What could be the bottleneck here?
Could it be qemu-img utility?
vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
/usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
-O raw
/rhev/data-center/mnt/glusterSD/10.20.30.41:
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
Any idea how to make it work faster or what throughput should I expected?
gluster storage operations are using fuse mount - so every write:
- travel to the kernel
- travel back to the gluster fuse helper process
- travel to all 3 replicas - replication is done on client side
- return to kernel when all writes succeeded
- return to caller
So gluster will never set any speed record.
Additionally, you are copying from raw lv on FC - qemu-img cannot do
anything
smart and avoid copying unused clusters. Instead if copies gigabytes of
zeros
from FC.
However 7.5-10 MiB/s sounds too slow.
I would try to test with dd - how much time it takes to copy
the same image from FC to your gluster storage?
dd
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
bs=8M oflag=direct status=progress
If dd can do this faster, please ask on qemu-discuss mailing list:
https://lists.nongnu.org/mailman/listinfo/qemu-discuss
If both give similar results, I think asking in gluster mailing list
about this can help. Maybe your gluster setup can be optimized.
Nir
Cheers,
Jiri
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
5:26 a.m.
This is a cryptographically signed message in MIME format.
--------------ms040706070909040905040207
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote:
On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka
<jiri.s=
lezka(a)slu.cz
<mailto:jiri.slezka@slu.cz>> wrote:
=20
Hi,
=20
I am trying realize why is exporting of vm to export storage on
glusterfs such slow.
=20
I am using oVirt and RHV, both instalations on version 4.1.7.
=20
Hosts have dedicated nics for rhevm network - 1gbps, data storage i=
tself
is on FC.
=20
GlusterFS cluster lives separate on 4 dedicated hosts. It has slow =
disks
but I can achieve about 200-400mbit throughput in other
application=
s (we
are using it for "cold" data, backups mostly).
=20
I am using this glusterfs cluster as backend for export storage. Wh=
en I
am exporting vm I can see only about 60-80mbit throughput.
=20
What could be the bottleneck here?
=20
Could it be qemu-img utility?
=20
vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0 0.0 354212 29148 ?=C2=
=A0
=C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A00:06
/usr/bin/qemu-img convert -p -t none -T none -f raw
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-100=
5-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57a=
cd5f-d6cf-48cc-ad0c-4a7d979c0c1e
-O raw
/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-a=
392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c5=
7acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
=20
Any idea how to make it work faster or what throughput should I
expected?
=20
=20
gluster storage operations are using fuse mount - so every write:
- travel to the kernel
- travel back to the gluster fuse helper process
- travel to all 3 replicas - replication is done on client side
- return to kernel when all writes succeeded
- return to caller
=20
So gluster will never set any speed record.
=20
Additionally, you are copying from raw lv on FC - qemu-img cannot do
anything
smart and avoid copying unused clusters. Instead if copies gigabytes of=
zeros
from FC.
ok, it does make sense
However 7.5-10 MiB/s sounds too slow.
=20
I would try to test with dd - how much time it takes to copy
the same image from FC to your gluster storage?
=20
dd
if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-10=
05-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57=
acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-=
a392-4ea2-b081-7c6288fbb636/__test__
bs=3D8M oflag=3Ddirect status=3Dprogress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
If dd can do this faster, please ask on qemu-discuss mailing list:
https://lists.nongnu.org/mailman/listinfo/qemu-discuss
=20
If both give similar results, I think asking in gluster mailing list
about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share=
=2E
Cheers,
Jiri
=20
Nir
=C2=A0
=20
=20
Cheers,
=20
Jiri
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
=20
--------------ms040706070909040905040207
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx
MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I
b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd
VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty
qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+
0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim
atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg
oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj
ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB
AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw
AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB
LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr
BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj
egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI
hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF
OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS
WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB
iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101
S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow
ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN
BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw
HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y
ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE
BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT
E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN
MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E
8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX
aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh
tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf
BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V
jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow
DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE
LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f
BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy
c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj
aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw
Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC
AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf
/SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx
VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts
UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd
M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB
MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB
bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl
cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG
SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMzExMjY0M1owLwYJ
KoZIhvcNAQkEMSIEIHDYuAnjwPWpgh4XDFXPeS0Y1CjWUEbKw7qRlk6OR0hFMGwGCSqGSIb3
DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG
AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES
MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl
U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC
CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE
BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j
ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBADEU
mf5wA7QyZ+c5nhjrS/RrBsua5NDZ+vk9/0bOUu17nJx17wOHKcFuqjTZV1kptc1LB0GrZpsE
GBBbtSYYlpTER54p3EDaV9LowE570S+kEUGyErANpyJmUIdJx1w9cPlIGBK9Kw23scDgnfy/
W21EWJyjlsIauzQ61tRl5NyZS9K1zOytQciB1lm/ac8XIaaHRq5VXolkLYWBW5HmBZoNifGv
M059/+cH+kzYD3zb/Py6/hbYbZEawz0f5W2dAMIxU1DgX7aJGJ2gz83WfCZBI1j7GE58y9Ww
DGcrZ9ALCch3Il+uGKCDvmb8kSqIqaT1rCbILkA2nvSAIC2qSjcAAAAAAAA=
--------------ms040706070909040905040207--
5:43 a.m.
This is a cryptographically signed message in MIME format.
--------------ms060403050704060409050400
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
well, another idea
when I did not use the direct flag, the performace was much better
15787360256 bytes (16 GB) copied, 422.955159 s, 37.3 MB/s
probably qemu-img uses direct write too and I understand why. But in
case of backup it is not as hot I think. Is there a chance to modify
this behavior for backup case? Is it a good idea? Should I fill RFE?
Cheers,
Jiri
On 11/23/2017 12:26 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka wrote:
Hi,
=20
On 11/22/2017 07:30 PM, Nir Soffer wrote:
> On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <jiri.=
slezka(a)slu.cz
> <mailto:jiri.slezka@slu.cz>> wrote:
>
> Hi,
>
> I am trying realize why is exporting of vm to export storage on
> glusterfs such slow.
>
> I am using oVirt and RHV, both instalations on version 4.1.7.
>
> Hosts have dedicated nics for rhevm network - 1gbps, data storage =
itself
> is on FC.
>
> GlusterFS cluster lives separate on 4 dedicated hosts. It has slow=
disks
> but I can achieve about 200-400mbit throughput in other
applicatio=
ns (we
> are using it for "cold" data, backups mostly).
>
> I am using this glusterfs cluster as backend for export storage. W=
hen I
> am exporting vm I can see only about 60-80mbit throughput.
>
> What could be the bottleneck here?
>
> Could it be qemu-img utility?
>
> vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0 0.0 354212 29148 ?=C2=
=A0
=C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A00:06
> /usr/bin/qemu-img convert -p -t none -T none -f raw
> /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-10=
05-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57=
acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> -O raw
> /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499-=
a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c=
57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>
> Any idea how to make it work faster or what throughput should I
> expected?
>
>
> gluster storage operations are using fuse mount - so every write:
> - travel to the kernel
> - travel back to the gluster fuse helper process
> - travel to all 3 replicas - replication is done on client side
> - return to kernel when all writes succeeded
> - return to caller
>
> So gluster will never set any speed record.
>
> Additionally, you are copying from raw lv on FC - qemu-img cannot do
> anything
> smart and avoid copying unused clusters. Instead if copies gigabytes o=
f
> zeros
> from FC.
=20
ok, it does make sense
=20
> However 7.5-10 MiB/s sounds too slow.
>
> I would try to test with dd - how much time it takes to copy
> the same image from FC to your gluster storage?
>
> dd
> if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1=
005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c5=
7acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>
of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/81094499=
-a392-4ea2-b081-7c6288fbb636/__test__
> bs=3D8M oflag=3Ddirect status=3Dprogress
=20
unfrotunately dd performs the same
=20
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
=20
=20
> If dd can do this faster, please ask on qemu-discuss mailing list:
> https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>
> If both give similar results, I think asking in gluster mailing list
> about this can help. Maybe your gluster setup can be optimized.
=20
ok, this is definitly on the gluster side. Thanks for your guidance.
=20
I will investigate the gluster side and also will try Export on NFS sha=
re.
=20
Cheers,
=20
Jiri
=20
=20
>
> Nir
> =C2=A0
>
>
> Cheers,
>
> Jiri
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
=20
=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
=20
--------------ms060403050704060409050400
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx
MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I
b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd
VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty
qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+
0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim
atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg
oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj
ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB
AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw
AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB
LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr
BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj
egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI
hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF
OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS
WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB
iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101
S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow
ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN
BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw
HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y
ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE
BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT
E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN
MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E
8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX
aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh
tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf
BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V
jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow
DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE
LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f
BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy
c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj
aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw
Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC
AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf
/SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx
VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts
UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd
M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB
MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB
bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl
cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG
SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyMzExNDM0NlowLwYJ
KoZIhvcNAQkEMSIEIMM2KSwC5xIOAF72J62BEBxPnPElG/qlnUVvTlWIGvR0MGwGCSqGSIb3
DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG
AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES
MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl
U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC
CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE
BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j
ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAK1V
y3x22QWoL0uYr5QPl+vYfp/UYzk7ngzeGaFUqIL8+SHBWDNuPINgjcHPo4Y5hxjnJ9i9sEVJ
qJ/dd/9/PorxVOaZh+ejg2lPw/3fCa7hJmCAt3U6yb1OdkqEVGSAfrCnJHXOlMMBbX/RNODk
ooTOGmItuhc8rzdSI5uOTOC/vpXdI4H8nbkzEUNpEADZO9a7wjjKTUezxhQ7TjE2SEBtTqay
gqP49+bTZb74O5BqZ352Y5QF7I1AAAV8U3RfIX7mtDlIr6Qhpyv4BSD5uo0vm45lOdoOhjvq
R7jshe+gfoyUTDCqeQ+ucQZRBvaAVg1L4IQQmBExo4ezhAnBMHAAAAAAAAA=
--------------ms060403050704060409050400--
7:32 a.m.
On Thu, Nov 23, 2017 at 1:43 PM, Jiří Sléžka <jiri.slezka(a)slu.cz> wrote:
well, another idea
when I did not use the direct flag, the performace was much better
15787360256 bytes (16 GB) copied, 422.955159 s, 37.3 MB/s
That means you were hitting the cache.
probably qemu-img uses direct write too and I understand why. But in
case of backup it is not as hot I think. Is there a chance to modify
this behavior for backup case? Is it a good idea? Should I fill RFE?
Probably not. We really prefer direct IO to ensure data is consistent.
Y.
Cheers,
Jiri
On 11/23/2017 12:26 PM, Jiří Sléžka wrote:
> Hi,
>
> On 11/22/2017 07:30 PM, Nir Soffer wrote:
>> On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka(a)slu.cz
>> <mailto:jiri.slezka@slu.cz>> wrote:
>>
>> Hi,
>>
>> I am trying realize why is exporting of vm to export storage on
>> glusterfs such slow.
>>
>> I am using oVirt and RHV, both instalations on version 4.1.7.
>>
>> Hosts have dedicated nics for rhevm network - 1gbps, data storage
itself
>> is on FC.
>>
>> GlusterFS cluster lives separate on 4 dedicated hosts. It has slow
disks
>> but I can achieve about 200-400mbit throughput in other
applications (we
>> are using it for "cold" data, backups mostly).
>>
>> I am using this glusterfs cluster as backend for export storage.
When I
>> am exporting vm I can see only about 60-80mbit throughput.
>>
>> What could be the bottleneck here?
>>
>> Could it be qemu-img utility?
>>
>> vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
>> /usr/bin/qemu-img convert -p -t none -T none -f raw
>> /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> -O raw
>> /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/images/
ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>>
>> Any idea how to make it work faster or what throughput should I
>> expected?
>>
>>
>> gluster storage operations are using fuse mount - so every write:
>> - travel to the kernel
>> - travel back to the gluster fuse helper process
>> - travel to all 3 replicas - replication is done on client side
>> - return to kernel when all writes succeeded
>> - return to caller
>>
>> So gluster will never set any speed record.
>>
>> Additionally, you are copying from raw lv on FC - qemu-img cannot do
>> anything
>> smart and avoid copying unused clusters. Instead if copies gigabytes of
>> zeros
>> from FC.
>
> ok, it does make sense
>
>> However 7.5-10 MiB/s sounds too slow.
>>
>> I would try to test with dd - how much time it takes to copy
>> the same image from FC to your gluster storage?
>>
>> dd
>> if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
>> bs=8M oflag=direct status=progress
>
> unfrotunately dd performs the same
>
> 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>
>
>> If dd can do this faster, please ask on qemu-discuss mailing list:
>> https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>>
>> If both give similar results, I think asking in gluster mailing list
>> about this can help. Maybe your gluster setup can be optimized.
>
> ok, this is definitly on the gluster side. Thanks for your guidance.
>
> I will investigate the gluster side and also will try Export on NFS
share.
>
> Cheers,
>
> Jiri
>
>
>>
>> Nir
>>
>>
>>
>> Cheers,
>>
>> Jiri
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:45 p.m.
On Thu, Nov 23, 2017 at 3:33 PM Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Thu, Nov 23, 2017 at 1:43 PM, Jiří Sléžka
<jiri.slezka(a)slu.cz> wrote:
> well, another idea
>
> when I did not use the direct flag, the performace was much better
>
> 15787360256 bytes (16 GB) copied, 422.955159 s, 37.3 MB/s
>
That means you were hitting the cache.
>
> probably qemu-img uses direct write too and I understand why. But in
> case of backup it is not as hot I think. Is there a chance to modify
> this behavior for backup case? Is it a good idea? Should I fill RFE?
>
Probably not. We really prefer direct IO to ensure data is consistent.
Y
I did some research in prehistoric oVirt source, and found why we started
to
use direct I/O. We have two issues:
1. reading stale data from storage
2. trashing host cache
If you don't use direct I/O when accessing shared storage, you risk reading
stale data from the kernel buffer cache. This cache may be stale since the
kernel
does not know anything about other hosts writing to the same storage after
the
last read from this storage.
The -t none option in vdsm was introduced because of
https://bugzilla.redhat.com/699976.
The qemu bug https://bugzilla.redhat.com/713743 explains the issue:
qemu-img was writing disk images using writeback and fillingup the cache
buffers
which are then flushed by the kernel preventing other processes from
accessing
the storage. This is particularly bad in cluster environments where
time-based
algorithms might be in place and accessing the storage within certain
timeouts
is critical
I'm not sure it this issue relevant now. We use now sanlock instead of
safelease,
(except for export domain still using safelease), and qemu or kernel may
have
better options to avoid trashing the host cache, or guarantee reliable
access
to storage.
Daivd, do you know if sanlock is effected by trashing the host cache?
Adding also qemu-block mailing list.
Nir
>
> Cheers,
>
> Jiri
>
>
> On 11/23/2017 12:26 PM, Jiří Sléžka wrote:
> > Hi,
> >
> > On 11/22/2017 07:30 PM, Nir Soffer wrote:
> >> On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka(a)slu.cz
> >> <mailto:jiri.slezka@slu.cz>> wrote:
> >>
> >> Hi,
> >>
> >> I am trying realize why is exporting of vm to export storage on
> >> glusterfs such slow.
> >>
> >> I am using oVirt and RHV, both instalations on version 4.1.7.
> >>
> >> Hosts have dedicated nics for rhevm network - 1gbps, data storage
> itself
> >> is on FC.
> >>
> >> GlusterFS cluster lives separate on 4 dedicated hosts. It has slow
> disks
> >> but I can achieve about 200-400mbit throughput in other
> applications (we
> >> are using it for "cold" data, backups mostly).
> >>
> >> I am using this glusterfs cluster as backend for export storage.
> When I
> >> am exporting vm I can see only about 60-80mbit throughput.
> >>
> >> What could be the bottleneck here?
> >>
> >> Could it be qemu-img utility?
> >>
> >> vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
> >> /usr/bin/qemu-img convert -p -t none -T none -f raw
> >>
>
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >> -O raw
> >> /rhev/data-center/mnt/glusterSD/10.20.30.41:
>
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >>
> >> Any idea how to make it work faster or what throughput should I
> >> expected?
> >>
> >>
> >> gluster storage operations are using fuse mount - so every write:
> >> - travel to the kernel
> >> - travel back to the gluster fuse helper process
> >> - travel to all 3 replicas - replication is done on client side
> >> - return to kernel when all writes succeeded
> >> - return to caller
> >>
> >> So gluster will never set any speed record.
> >>
> >> Additionally, you are copying from raw lv on FC - qemu-img cannot do
> >> anything
> >> smart and avoid copying unused clusters. Instead if copies gigabytes of
> >> zeros
> >> from FC.
> >
> > ok, it does make sense
> >
> >> However 7.5-10 MiB/s sounds too slow.
> >>
> >> I would try to test with dd - how much time it takes to copy
> >> the same image from FC to your gluster storage?
> >>
> >> dd
> >>
>
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >> of=/rhev/data-center/mnt/glusterSD/10.20.30.41:
> _rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> >> bs=8M oflag=direct status=progress
> >
> > unfrotunately dd performs the same
> >
> > 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
> >
> >
> >> If dd can do this faster, please ask on qemu-discuss mailing list:
> >> https://lists.nongnu.org/mailman/listinfo/qemu-discuss
> >>
> >> If both give similar results, I think asking in gluster mailing list
> >> about this can help. Maybe your gluster setup can be optimized.
> >
> > ok, this is definitly on the gluster side. Thanks for your guidance.
> >
> > I will investigate the gluster side and also will try Export on NFS
> share.
> >
> > Cheers,
> >
> > Jiri
> >
> >
> >>
> >> Nir
> >>
> >>
> >>
> >> Cheers,
> >>
> >> Jiri
> >>
> >>
> >> _______________________________________________
> >> Users mailing list
> >> Users(a)ovirt.org <mailto:Users@ovirt.org>
> >> http://lists.ovirt.org/mailman/listinfo/users
> >>
> >
> >
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
> _______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:18 a.m.
New subject: [Qemu-block] slow performance with export storage on glusterfs
Am 07.12.2017 um 23:45 hat Nir Soffer geschrieben:
The qemu bug https://bugzilla.redhat.com/713743 explains the issue:
qemu-img was writing disk images using writeback and fillingup the
cache buffers which are then flushed by the kernel preventing other
processes from accessing the storage. This is particularly bad in
cluster environments where time-based algorithms might be in place and
accessing the storage within certain timeouts is critical
I'm not sure it this issue relevant now. We use now sanlock instead of
safelease, (except for export domain still using safelease), and qemu
or kernel may have better options to avoid trashing the host cache, or
guarantee reliable access to storage.
Non-direct means that the data goes through the kernel page cache, and
the kernel doesn't know that it won't be needed again, so it will fill
up the cache with the image.
I'm also not aware that cache coherency is now provided by all backends
for shared storage, so O_DIRECT still seems to be the only way to avoid
using stale caches. Since the problem is about stale caches, I don't see
how the locking mechanism could make a difference.
The only thing I can suggest, given that there is a "glusterfs" in the
subject line of the email, is that the native gluster driver in QEMU
takes a completely different path and never uses the kernel page cache,
which should make both problems disappear. Maybe it would be worth
having a look at this.
Kevin
8:42 a.m.
New subject: [Qemu-block] slow performance with export storage on glusterfs
On Fri, Dec 8, 2017 at 4:18 PM Kevin Wolf <kwolf(a)redhat.com> wrote:
Am 07.12.2017 um 23:45 hat Nir Soffer geschrieben:
> The qemu bug https://bugzilla.redhat.com/713743 explains the issue:
> qemu-img was writing disk images using writeback and fillingup the
> cache buffers which are then flushed by the kernel preventing other
> processes from accessing the storage. This is particularly bad in
> cluster environments where time-based algorithms might be in place and
> accessing the storage within certain timeouts is critical
>
> I'm not sure it this issue relevant now. We use now sanlock instead of
> safelease, (except for export domain still using safelease), and qemu
> or kernel may have better options to avoid trashing the host cache, or
> guarantee reliable access to storage.
Non-direct means that the data goes through the kernel page cache, and
the kernel doesn't know that it won't be needed again, so it will fill
up the cache with the image.
I'm also not aware that cache coherency is now provided by all backends
for shared storage, so O_DIRECT still seems to be the only way to avoid
using stale caches. Since the problem is about stale caches, I don't see
how the locking mechanism could make a difference.
The only thing I can suggest, given that there is a "glusterfs" in the
subject line of the email, is that the native gluster driver in QEMU
takes a completely different path and never uses the kernel page cache,
which should make both problems disappear. Maybe it would be worth
having a look at this.
This is an interesting direction - currently oVirt does not use native
glusterfs for qemu-img operation, only for running vms. We still use
the fuse based glusterfs mount for storage operations like copying
and converting images.
Kevin
11:41 p.m.
On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka(a)slu.cz> wrote:
Hi,
On 11/22/2017 07:30 PM, Nir Soffer wrote:
> On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka(a)slu.cz
> <mailto:jiri.slezka@slu.cz>> wrote:
>
> Hi,
>
> I am trying realize why is exporting of vm to export storage on
> glusterfs such slow.
>
> I am using oVirt and RHV, both instalations on version 4.1.7.
>
> Hosts have dedicated nics for rhevm network - 1gbps, data storage
itself
> is on FC.
>
> GlusterFS cluster lives separate on 4 dedicated hosts. It has slow
disks
> but I can achieve about 200-400mbit throughput in other applications
(we
> are using it for "cold" data, backups mostly).
>
> I am using this glusterfs cluster as backend for export storage.
When I
> am exporting vm I can see only about 60-80mbit throughput.
>
> What could be the bottleneck here?
>
> Could it be qemu-img utility?
>
> vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06
> /usr/bin/qemu-img convert -p -t none -T none -f raw
> /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> -O raw
> /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/images/
ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>
> Any idea how to make it work faster or what throughput should I
> expected?
>
>
> gluster storage operations are using fuse mount - so every write:
> - travel to the kernel
> - travel back to the gluster fuse helper process
> - travel to all 3 replicas - replication is done on client side
> - return to kernel when all writes succeeded
> - return to caller
>
> So gluster will never set any speed record.
>
> Additionally, you are copying from raw lv on FC - qemu-img cannot do
> anything
> smart and avoid copying unused clusters. Instead if copies gigabytes of
> zeros
> from FC.
ok, it does make sense
> However 7.5-10 MiB/s sounds too slow.
>
> I would try to test with dd - how much time it takes to copy
> the same image from FC to your gluster storage?
>
> dd
> if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=8M oflag=direct status=progress
unfrotunately dd performs the same
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
> If dd can do this faster, please ask on qemu-discuss mailing list:
> https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>
> If both give similar results, I think asking in gluster mailing list
> about this can help. Maybe your gluster setup can be optimized.
ok, this is definitly on the gluster side. Thanks for your guidance.
I will investigate the gluster side and also will try Export on NFS share.
[Adding gluster users ml]
Please provide "gluster volume info" output for the rhv_export gluster
volume and also volume profile details (refer to earlier mail from Shani on
how to run this) while performing the dd operation above.
Cheers,
Jiri
>
> Nir
>
>
>
> Cheers,
>
> Jiri
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
2:48 a.m.
This is a cryptographically signed message in MIME format.
--------------ms000801020502000000010003
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
On 11/24/2017 06:41 AM, Sahina Bose wrote:
=20
=20
On Thu, Nov 23, 2017 at 4:56 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <jiri.=
slezka(a)slu.cz
<mailto:jiri.slezka@slu.cz>> wrote:
=20
Hi,
=20
On 11/22/2017 07:30 PM, Nir Soffer wrote:
> On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <=
jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>
> <mailto:jiri.slezka@slu.cz
<mailto:jiri.slezka@slu.cz>>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0Hi,
>
>=C2=A0 =C2=A0 =C2=A0I am trying realize why is exporting of vm to =
export
storage on
>=C2=A0 =C2=A0 =C2=A0glusterfs such slow.
>
>=C2=A0 =C2=A0 =C2=A0I am using oVirt and RHV, both instalations on=
version 4.1.7.
>
>=C2=A0 =C2=A0 =C2=A0Hosts have dedicated nics for rhevm network - =
1gbps,
data
storage itself
>=C2=A0 =C2=A0 =C2=A0is on FC.
>
>=C2=A0 =C2=A0 =C2=A0GlusterFS cluster lives separate on 4 dedicate=
d
hosts. It has
slow disks
>=C2=A0 =C2=A0 =C2=A0but I can achieve about 200-400mbit throughput=
in
other
applications (we
>=C2=A0 =C2=A0 =C2=A0are using it for "cold" data, backups mostly).=
>
>=C2=A0 =C2=A0 =C2=A0I am using this glusterfs cluster as backend f=
or
export
storage. When I
>=C2=A0 =C2=A0 =C2=A0am exporting vm I can see only about 60-80mbit=
throughput.
>
>=C2=A0 =C2=A0 =C2=A0What could be the bottleneck here?
>
>=C2=A0 =C2=A0 =C2=A0Could it be qemu-img utility?
>
>=C2=A0 =C2=A0 =C2=A0vdsm=C2=A0 =C2=A0 =C2=A0 97739=C2=A0 0.3=C2=A0=
0.0
354212 29148 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 S<l=C2=A0 15:43=C2=A0 =C2=A0=
0:06
>=C2=A0 =C2=A0 =C2=A0/usr/bin/qemu-img convert -p -t none -T
none -=
f raw
>=C2=A0 =C2=A0
=C2=A0/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd5=
6a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e2=
7/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0-O raw
>=C2=A0 =C2=A0
=C2=A0/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/8109=
4499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1=
e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>
>=C2=A0 =C2=A0 =C2=A0Any idea how to make it work faster or what th=
roughput should I
>=C2=A0 =C2=A0 =C2=A0expected?
>
>
> gluster storage operations are using fuse mount - so every write:=
> - travel to the kernel
> - travel back to the gluster fuse helper process
> - travel to all 3 replicas - replication is done on client side
> - return to kernel when all writes succeeded
> - return to caller
>
> So gluster will never set any speed record.
>
> Additionally, you are copying from raw lv on FC - qemu-img cannot=
do
> anything
> smart and avoid copying unused clusters. Instead if copies
gigabytes of
> zeros
> from FC.
=20
ok, it does make sense
=20
> However 7.5-10 MiB/s sounds too slow.
>
> I would try to test with dd - how much time it takes to copy
> the same image from FC to your gluster storage?
>
> dd
> if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd=
56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e=
27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>
of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/810=
94499-a392-4ea2-b081-7c6288fbb636/__test__
> bs=3D8M oflag=3Ddirect status=3Dprogress
=20
unfrotunately dd performs the same
=20
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
=20
=20
> If dd can do this faster, please ask on qemu-discuss mailing list=
:
> https://lists.nongnu.org/mailman/listinfo/qemu-discuss
<https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
>
> If both give similar results, I think asking in gluster mailing l=
ist
> about this can help. Maybe your gluster setup can be
optimized.
=20
ok, this is definitly on the gluster side. Thanks for your guidance=
=2E
=20
I will investigate the gluster side and also will try Export on NFS=
share.
=20
=20
[Adding gluster users ml]
=20
Please provide "gluster volume info" output for the rhv_export gluster
volume and also volume profile details (refer to earlier mail from Shan=
i
on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8
as mentioned in other posts. Gluster cluster uses really slow (green)
disks but without direct io it can achieve throughput around 400mbit/s.
This storage is used mostly for backup purposes. It is not used as a vm
storage.
In my case it would be nice not to use direct io in export case but I
understand why it might not be wise.
Cheers,
Jiri
=20
=C2=A0
=20
=20
Cheers,
=20
Jiri
=20
=20
>
> Nir
> =C2=A0
>
>
>=C2=A0 =C2=A0 =C2=A0Cheers,
>
>=C2=A0 =C2=A0 =C2=A0Jiri
>
>
>=C2=A0 =C2=A0 =C2=A0______________________________________________=
_
>=C2=A0 =C2=A0 =C2=A0Users mailing list
>=C2=A0 =C2=A0 =C2=A0Users(a)ovirt.org <mailto:Users@ovirt.org>
<mailto:Users@ovirt.org <mailto:Users@ovirt.org>>
>=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
>
=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
--------------ms000801020502000000010003
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx
MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I
b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd
VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty
qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+
0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim
atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg
oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj
ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB
AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw
AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB
LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr
BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj
egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI
hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF
OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS
WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB
iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101
S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow
ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN
BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw
HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y
ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE
BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT
E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN
MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E
8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX
aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh
tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf
BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V
jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow
DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE
LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f
BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy
c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj
aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw
Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC
AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf
/SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx
VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts
UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd
M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB
MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB
bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl
cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG
SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyNDA4NDgyOVowLwYJ
KoZIhvcNAQkEMSIEICc3XTVcQ/q6/38okZAzf1u/2L2Kvnr7lC0WPziYo539MGwGCSqGSIb3
DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG
AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES
MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl
U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC
CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE
BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j
ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBADBi
5DSVYHZZJJKx30+0/xl2OX/7btvMHwfEOD4uIYjTxA2PXjZztLBcyXV7+vPX/SJzGTz38SH5
/FvzDOVXAAaaF0X9Yf+LHLUT1yEY/oc8Mabjbwkg8GzIxUkuTeSoidGnaGaCO26X2sD+Txq/
HZYi0oVqrSVN045GHYi+sBDh7t6QnOB9D/ckidlajCl2iZ2bOe/q19aisAJcP1VrH5H0Mtsc
YHS/SlwN32PwO6kS/JZuKVjn8PPiVh6yKeNiESxcMAVbhybq+fGAvn9UZcpcAOzs/8qleVLK
3RT33LCh1QH1COGuI/KeXRCjN/LRBzJ/2w2BDOOg5/zFggSoaFEAAAAAAAA=
--------------ms000801020502000000010003--
7:41 p.m.
What about mounting over nfs instead of the fuse client. Or maybe libgfapi.
Is that available for export domains
On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka(a)slu.cz> wrote:
On 11/24/2017 06:41 AM, Sahina Bose wrote:
>
>
> On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka(a)slu.cz
> <mailto:jiri.slezka@slu.cz>> wrote:
>
> Hi,
>
> On 11/22/2017 07:30 PM, Nir Soffer wrote:
> > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka(a)slu.cz
<mailto:jiri.slezka@slu.cz>
> > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>
wrote:
> >
> > Hi,
> >
> > I am trying realize why is exporting of vm to export storage on
> > glusterfs such slow.
> >
> > I am using oVirt and RHV, both instalations on version 4.1.7.
> >
> > Hosts have dedicated nics for rhevm network - 1gbps, data
> storage itself
> > is on FC.
> >
> > GlusterFS cluster lives separate on 4 dedicated hosts. It has
> slow disks
> > but I can achieve about 200-400mbit throughput in other
> applications (we
> > are using it for "cold" data, backups mostly).
> >
> > I am using this glusterfs cluster as backend for export
> storage. When I
> > am exporting vm I can see only about 60-80mbit throughput.
> >
> > What could be the bottleneck here?
> >
> > Could it be qemu-img utility?
> >
> > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43
0:06
> > /usr/bin/qemu-img convert -p -t none -T none -f raw
> >
>
/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > -O raw
> >
> /rhev/data-center/mnt/glusterSD/10.20.30.41:
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> >
> > Any idea how to make it work faster or what throughput should I
> > expected?
> >
> >
> > gluster storage operations are using fuse mount - so every write:
> > - travel to the kernel
> > - travel back to the gluster fuse helper process
> > - travel to all 3 replicas - replication is done on client side
> > - return to kernel when all writes succeeded
> > - return to caller
> >
> > So gluster will never set any speed record.
> >
> > Additionally, you are copying from raw lv on FC - qemu-img cannot
do
> > anything
> > smart and avoid copying unused clusters. Instead if copies
> gigabytes of
> > zeros
> > from FC.
>
> ok, it does make sense
>
> > However 7.5-10 MiB/s sounds too slow.
> >
> > I would try to test with dd - how much time it takes to copy
> > the same image from FC to your gluster storage?
> >
> > dd
> >
if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:
_rhv__export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> > bs=8M oflag=direct status=progress
>
> unfrotunately dd performs the same
>
> 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>
>
> > If dd can do this faster, please ask on qemu-discuss mailing list:
> > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
> <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
> >
> > If both give similar results, I think asking in gluster mailing
list
> > about this can help. Maybe your gluster setup can be optimized.
>
> ok, this is definitly on the gluster side. Thanks for your guidance.
>
> I will investigate the gluster side and also will try Export on NFS
> share.
>
>
> [Adding gluster users ml]
>
> Please provide "gluster volume info" output for the rhv_export gluster
> volume and also volume profile details (refer to earlier mail from Shani
> on how to run this) while performing the dd operation above.
you can find all this output on https://pastebin.com/sBK01VS8
as mentioned in other posts. Gluster cluster uses really slow (green)
disks but without direct io it can achieve throughput around 400mbit/s.
This storage is used mostly for backup purposes. It is not used as a vm
storage.
In my case it would be nice not to use direct io in export case but I
understand why it might not be wise.
Cheers,
Jiri
>
>
>
>
> Cheers,
>
> Jiri
>
>
> >
> > Nir
> >
> >
> >
> > Cheers,
> >
> > Jiri
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org <mailto:Users@ovirt.org>
> <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>
> > http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
> >
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
> <http://lists.ovirt.org/mailman/listinfo/users>
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:27 a.m.
New subject: [Gluster-users] slow performance with export storage on glusterfs
Hello
If you use Gluster as FUSE mount it's always slower than you expect it to
be.
If you want to get better performance out of your oVirt/Gluster storage,
try the following:
- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks
(Virtual disks are located on your Gluster storage volume).
- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
(lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
- then install NFS server and export LV you created in previous step, use
the NFS export as export domain in oVirt/RHEV.
You should get wire speed when you use multiple stripes on Gluster storage,
FUSE mount on oVirt host will fan out requests to all 4 servers.
Gluster is very good at distributed/parallel workloads, but when you use
direct Gluster FUSE mount for Export domain you only have one data stream,
which is fragmented even more my multiple writes/reads that Gluster needs
to do to save your data on all member servers.
On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny(a)fortnebula.com> wrote:
What about mounting over nfs instead of the fuse client. Or maybe
libgfapi. Is that available for export domains
On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.slezka(a)slu.cz> wrote:
> On 11/24/2017 06:41 AM, Sahina Bose wrote:
> >
> >
> > On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.slezka(a)slu.cz
> > <mailto:jiri.slezka@slu.cz>> wrote:
> >
> > Hi,
> >
> > On 11/22/2017 07:30 PM, Nir Soffer wrote:
> > > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.slezka(a)slu.cz
> <mailto:jiri.slezka@slu.cz>
> > > <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>
wrote:
> > >
> > > Hi,
> > >
> > > I am trying realize why is exporting of vm to export storage
> on
> > > glusterfs such slow.
> > >
> > > I am using oVirt and RHV, both instalations on version 4.1.7.
> > >
> > > Hosts have dedicated nics for rhevm network - 1gbps, data
> > storage itself
> > > is on FC.
> > >
> > > GlusterFS cluster lives separate on 4 dedicated hosts. It has
> > slow disks
> > > but I can achieve about 200-400mbit throughput in other
> > applications (we
> > > are using it for "cold" data, backups mostly).
> > >
> > > I am using this glusterfs cluster as backend for export
> > storage. When I
> > > am exporting vm I can see only about 60-80mbit throughput.
> > >
> > > What could be the bottleneck here?
> > >
> > > Could it be qemu-img utility?
> > >
> > > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43
> 0:06
> > > /usr/bin/qemu-img convert -p -t none -T none -f raw
> > >
> > /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > > -O raw
> > >
> > /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
> export/81094499-a392-4ea2-b081-7c6288fbb636/images/
> ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > >
> > > Any idea how to make it work faster or what throughput should
> I
> > > expected?
> > >
> > >
> > > gluster storage operations are using fuse mount - so every write:
> > > - travel to the kernel
> > > - travel back to the gluster fuse helper process
> > > - travel to all 3 replicas - replication is done on client side
> > > - return to kernel when all writes succeeded
> > > - return to caller
> > >
> > > So gluster will never set any speed record.
> > >
> > > Additionally, you are copying from raw lv on FC - qemu-img cannot
> do
> > > anything
> > > smart and avoid copying unused clusters. Instead if copies
> > gigabytes of
> > > zeros
> > > from FC.
> >
> > ok, it does make sense
> >
> > > However 7.5-10 MiB/s sounds too slow.
> > >
> > > I would try to test with dd - how much time it takes to copy
> > > the same image from FC to your gluster storage?
> > >
> > > dd
> > > if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
> > > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
> export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
> > > bs=8M oflag=direct status=progress
> >
> > unfrotunately dd performs the same
> >
> > 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
> >
> >
> > > If dd can do this faster, please ask on qemu-discuss mailing list:
> > > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
> > <https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
> > >
> > > If both give similar results, I think asking in gluster mailing
> list
> > > about this can help. Maybe your gluster setup can be optimized.
> >
> > ok, this is definitly on the gluster side. Thanks for your guidance.
> >
> > I will investigate the gluster side and also will try Export on NFS
> > share.
> >
> >
> > [Adding gluster users ml]
> >
> > Please provide "gluster volume info" output for the rhv_export
gluster
> > volume and also volume profile details (refer to earlier mail from Shani
> > on how to run this) while performing the dd operation above.
>
> you can find all this output on https://pastebin.com/sBK01VS8
>
> as mentioned in other posts. Gluster cluster uses really slow (green)
> disks but without direct io it can achieve throughput around 400mbit/s.
>
> This storage is used mostly for backup purposes. It is not used as a vm
> storage.
>
> In my case it would be nice not to use direct io in export case but I
> understand why it might not be wise.
>
> Cheers,
>
> Jiri
>
> >
> >
> >
> >
> > Cheers,
> >
> > Jiri
> >
> >
> > >
> > > Nir
> > >
> > >
> > >
> > > Cheers,
> > >
> > > Jiri
> > >
> > >
> > > _______________________________________________
> > > Users mailing list
> > > Users(a)ovirt.org <mailto:Users@ovirt.org>
> > <mailto:Users@ovirt.org <mailto:Users@ovirt.org>>
> > > http://lists.ovirt.org/mailman/listinfo/users
> > <http://lists.ovirt.org/mailman/listinfo/users>
> > >
> >
> >
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org <mailto:Users@ovirt.org>
> > http://lists.ovirt.org/mailman/listinfo/users
> > <http://lists.ovirt.org/mailman/listinfo/users>
> >
> >
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Gluster-users mailing list
Gluster-users(a)gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
3:08 a.m.
New subject: [Gluster-users] slow performance with export storage on glusterfs
This is a cryptographically signed message in MIME format.
--------------ms050302080405030009040507
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Hello,
=20
If you use Gluster as FUSE mount it's always slower than you expect it
to be.
If you want to get better performance out of your oVirt/Gluster storage=
,
try the following:=C2=A0
=20
- create a Linux VM in your oVirt environment, assign 4/8/12 virtual
disks (Virtual disks are located on your Gluster storage volume).
- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
(lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
- then install NFS server and export LV you created in previous step,
use the NFS export as export domain in oVirt/RHEV.
=20
You should get wire speed when you use multiple stripes on Gluster
storage, FUSE mount on oVirt host will fan out requests to all 4 server=
s.
Gluster is very good at distributed/parallel workloads, but when you
us=
e
direct Gluster FUSE mount for Export domain you only have one data
stream, which is fragmented even more my multiple writes/reads that
Gluster needs to do to save your data on all member servers.
Thanks for explanation, it is an interesting solution.
Cheers,
Jiri
=20
=20
=20
On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <donny(a)fortnebula.com
<mailto:donny@fortnebula.com>> wrote:
=20
What about mounting over nfs instead of the fuse client. Or maybe
libgfapi. Is that available for export domains
=20
On Fri, Nov 24, 2017 at 3:48 AM Ji=C5=99=C3=AD Sl=C3=A9=C5=BEka <ji=
ri.slezka(a)slu.cz
<mailto:jiri.slezka@slu.cz>> wrote:
=20
On 11/24/2017 06:41 AM, Sahina Bose wrote:
>
>
> On Thu, Nov 23, 2017 at 4:56 PM, Ji=C5=99=C3=AD Sl=C3=A9=C5=BE=
ka
<jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>
> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>
wrot=
e:
>
>=C2=A0 =C2=A0 =C2=A0Hi,
>
>=C2=A0 =C2=A0 =C2=A0On 11/22/2017 07:30 PM, Nir Soffer wrote:
>=C2=A0 =C2=A0 =C2=A0> On Mon, Nov 20, 2017 at 5:22 PM Ji=C5=99=
=C3=AD Sl=C3=A9=C5=BEka
<jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>
<mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>
>=C2=A0 =C2=A0 =C2=A0> <mailto:jiri.slezka@slu.cz <mailto:jiri.=
slezka(a)slu.cz>
<mailto:jiri.slezka@slu.cz
<mailto:jiri.slezka@slu.cz>>>> wrote=
:
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Hi,
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0I am trying realize w=
hy
is exporting of vm to export
storage on
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0glusterfs such slow.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0I am using oVirt and =
RHV,
both instalations on
version 4.1.7.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Hosts have dedicated =
nics
for rhevm network - 1gbps,
data
>=C2=A0 =C2=A0 =C2=A0storage itself
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0is on FC.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0GlusterFS cluster liv=
es
separate on 4 dedicated
hosts. It has
>=C2=A0 =C2=A0 =C2=A0slow disks
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0but I can achieve abo=
ut
200-400mbit throughput in other
>=C2=A0 =C2=A0 =C2=A0applications (we
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0are using it for "col=
d" data, backups mostly).
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0I am using this glust=
erfs
cluster as backend for export
>=C2=A0 =C2=A0 =C2=A0storage. When I
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0am exporting vm I can=
see
only about 60-80mbit
throughput.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0What could be the bot=
tleneck here?
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Could it be qemu-img =
utility?
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0vdsm=C2=A0 =C2=A0 =C2=
=A0
97739=C2=A0 0.3=C2=A0 0.0 354212 29148 ?=C2=A0 =C2=A0 =C2=A0 =C2=A0 S=
<l=C2=A0
15:43=C2=A0 =C2=A00:06
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0/usr/bin/qemu-img con=
vert
-p -t none -T none -f raw
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0
>=C2=A0 =C2=A0
=C2=A0=C2=A0/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945=
f5f/ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-0=
0f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0-O raw
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0
>=C2=A0 =C2=A0
=C2=A0=C2=A0/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__e=
xport/81094499-a392-4ea2-b081-7c6288fbb636/images/ba42cbcc-c068-4df8-af3d=
-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Any idea how to make =
it
work faster or what
throughput should I
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0expected?
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> gluster storage operations are using fus=
e
mount - so
every write:
>=C2=A0 =C2=A0 =C2=A0> - travel to the kernel
>=C2=A0 =C2=A0 =C2=A0> - travel back to the gluster fuse helper=
process
>=C2=A0 =C2=A0 =C2=A0> - travel to all 3 replicas -
replication=
is done on
client side
>=C2=A0 =C2=A0 =C2=A0> - return to kernel when all writes succe=
eded
>=C2=A0 =C2=A0 =C2=A0> - return to caller
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> So gluster will never set any speed reco=
rd.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> Additionally, you are copying from raw l=
v on
FC -
qemu-img cannot do
>=C2=A0 =C2=A0 =C2=A0> anything
>=C2=A0 =C2=A0 =C2=A0> smart and avoid copying unused clusters.=
Instead if copies
>=C2=A0 =C2=A0 =C2=A0gigabytes of
>=C2=A0 =C2=A0 =C2=A0> zeros
>=C2=A0 =C2=A0 =C2=A0> from FC.
>
>=C2=A0 =C2=A0 =C2=A0ok, it does make sense
>
>=C2=A0 =C2=A0 =C2=A0> However 7.5-10 MiB/s sounds too slow.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> I would try to test with dd - how much t=
ime
it takes to copy
>=C2=A0 =C2=A0 =C2=A0> the same image from FC to your
gluster s=
torage?
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> dd
>=C2=A0 =C2=A0 =C2=A0>
if=3D/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ff3=
cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-c068-4df8-af3d-00f2077b=
1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>=C2=A0 =C2=A0 =C2=A0>
of=3D/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__export/8=
1094499-a392-4ea2-b081-7c6288fbb636/__test__
>=C2=A0 =C2=A0 =C2=A0> bs=3D8M oflag=3Ddirect
status=3Dprogress=
>
>=C2=A0 =C2=A0 =C2=A0unfrotunately dd performs the same
>
>=C2=A0 =C2=A0 =C2=A01778384896 bytes (1.8 GB) copied, 198.5652=
65 s,
9.0 MB/s
>
>
>=C2=A0 =C2=A0 =C2=A0> If dd can do this faster, please ask on =
qemu-discuss
mailing list:
>=C2=A0 =C2=A0 =C2=A0> https://lists.nongnu.org/mailman/listinf=
o/qemu-discuss
<https://lists.nongnu.org/mailman/listinfo/qemu-discuss>
>=C2=A0 =C2=A0 =C2=A0<https://lists.nongnu.org/mailman/listinfo=
/qemu-discuss
<https://lists.nongnu.org/mailman/listinfo/qemu-discuss>>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> If both give similar results, I think as=
king
in gluster
mailing list
>=C2=A0 =C2=A0 =C2=A0> about this can help. Maybe your gluster =
setup can be
optimized.
>
>=C2=A0 =C2=A0 =C2=A0ok, this is definitly on the gluster side.=
Thanks
for your
guidance.
>
>=C2=A0 =C2=A0 =C2=A0I will investigate the gluster side and al=
so will
try
Export on NFS
>=C2=A0 =C2=A0 =C2=A0share.
>
>
> [Adding gluster users ml]
>
> Please provide "gluster volume info" output for the rhv_expor=
t
gluster
> volume and also volume profile details (refer to earlier mail=
from Shani
> on how to run this) while performing the dd operation above.
=20
you can find all this output on https://pastebin.com/sBK01VS8
=20
as mentioned in other posts. Gluster cluster uses really slow
(green)
disks but without direct io it can achieve throughput around
400mbit/s.
=20
This storage is used mostly for backup purposes. It is not used=
as a vm
storage.
=20
In my case it would be nice not to use direct io in export case=
but I
understand why it might not be wise.
=20
Cheers,
=20
Jiri
=20
>
> =C2=A0
>
>
>=C2=A0 =C2=A0 =C2=A0Cheers,
>
>=C2=A0 =C2=A0 =C2=A0Jiri
>
>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> Nir
>=C2=A0 =C2=A0 =C2=A0> =C2=A0
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Cheers,
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Jiri
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0_____________________=
__________________________
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Users mailing
list
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0Users(a)ovirt.org <mail=
to:Users@ovirt.org>
<mailto:Users@ovirt.org
<mailto:Users@ovirt.org>>
>=C2=A0 =C2=A0 =C2=A0<mailto:Users@ovirt.org <mailto:Users@ovir=
t.org>
<mailto:Users@ovirt.org
<mailto:Users@ovirt.org>>>
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.or=
g/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
>=C2=A0 =C2=A0 =C2=A0<http://lists.ovirt.org/mailman/listinfo/u=
sers
<http://lists.ovirt.org/mailman/listinfo/users>>
>=C2=A0 =C2=A0 =C2=A0>
>
>
>
>=C2=A0 =C2=A0 =C2=A0__________________________________________=
_____
>=C2=A0 =C2=A0 =C2=A0Users mailing list
>=C2=A0 =C2=A0 =C2=A0Users(a)ovirt.org <mailto:Users@ovirt.org>
<mailto:Users@ovirt.org <mailto:Users@ovirt.org>>
>=C2=A0 =C2=A0 =C2=A0http://lists.ovirt.org/mailman/listinfo/us=
ers
<http://lists.ovirt.org/mailman/listinfo/users>
>=C2=A0 =C2=A0 =C2=A0<http://lists.ovirt.org/mailman/listinfo/u=
sers
<http://lists.ovirt.org/mailman/listinfo/users>>
>
>
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
_______________________________________________
Gluster-users mailing list
Gluster-users(a)gluster.org <mailto:Gluster-users@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
=20
=20
--------------ms050302080405030009040507
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC
Cn8wggUJMIID8aADAgECAhACt8ndrdK9CetZxFyQDGB4MA0GCSqGSIb3DQEBCwUAMGUxCzAJ
BgNVBAYTAlVTMRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2Vy
dC5jb20xJDAiBgNVBAMTG0RpZ2lDZXJ0IEFzc3VyZWQgSUQgUm9vdCBDQTAeFw0xNDExMTgx
MjAwMDBaFw0yNDExMTgxMjAwMDBaMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1I
b2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMd
VEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwp9Jj5Aej1xPkS1GV3LvBdemFmkUR//nSzBodqsU3dv2BCRD30r4gt5oRsYty
qDGF2nnItxV1SkwVoDxFeRzOIHYNYvBRHaiGvCQjEXzPRTocOSVfWpmq/zAL/QOEqpJogeM+
0IBGiJcAENJshl7UcfjYbBnN5qStk74f52VWFf/aiF7MVJnsUr3oriQvXYOzs8N/NXyyQyim
atBbumJVCNszF1X+XHCGfPNvxlNFW9ktv7azK0baminfLcsh6ubCdINZc+Nof2lU387NCDgg
oh3KsYVcZTSuhh7qp6MjxE5VqOZod1hpXXzDOkjK+DAMC57iZXssncp24eaN08VlAgMBAAGj
ggGmMIIBojASBgNVHRMBAf8ECDAGAQH/AgEAMA4GA1UdDwEB/wQEAwIBhjB5BggrBgEFBQcB
AQRtMGswJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBDBggrBgEFBQcw
AoY3aHR0cDovL2NhY2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENB
LmNydDCBgQYDVR0fBHoweDA6oDigNoY0aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lD
ZXJ0QXNzdXJlZElEUm9vdENBLmNybDA6oDigNoY0aHR0cDovL2NybDQuZGlnaWNlcnQuY29t
L0RpZ2lDZXJ0QXNzdXJlZElEUm9vdENBLmNybDA9BgNVHSAENjA0MDIGBFUdIAAwKjAoBggr
BgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAdBgNVHQ4EFgQUjJ8RLubj
egSlHlWLRggEpu2XcKYwHwYDVR0jBBgwFoAUReuir/SSy4IxLVGLp6chnfNtyA8wDQYJKoZI
hvcNAQELBQADggEBAI5HEV91Oen8WHFCoJkeu2Av+b/kWTV2qH/YNI1Xsbou2hHKhh4IyNkF
OxA/TUiuK2qQnQ5hAS0TIrs9SJ1Ke+DjXd/cTBiw7lCYSW5hkzigFV+iSivninpItafWqYBS
WxITl1KHBS9YBskhEqO5GLliDMPiAgjqUBQ/H1qZMlZNQIuFu0UaFUQuZUpJFr4+0zpzPxsB
iWU2muAoGItwbaP55EYshM7+v/J+x6kIhAJt5Dng8fOmOvR9F6Vw2/E0EZ6oQ8g1fdhwM101
S1OI6J1tUil1r7ES/svNqVWVb7YkUEBcPo8ppfHnTI/uxsn2tslsWefsOGJxNYUUSMAb9Eow
ggVuMIIEVqADAgECAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBCwUAMHIxCzAJBgNV
BAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlBbXN0ZXJkYW0xDzAN
BgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBlcnNvbmFsIENBIDMw
HhcNMTcxMTE2MDAwMDAwWhcNMTgxMjE1MTIwMDAwWjCBlDETMBEGCgmSJomT8ixkARkWA29y
ZzEWMBQGCgmSJomT8ixkARkWBnRlcmVuYTETMBEGCgmSJomT8ixkARkWA3RjczELMAkGA1UE
BhMCQ1oxJTAjBgNVBAoTHFNpbGVzaWFuIFVuaXZlcnNpdHkgaW4gT3BhdmExHDAaBgNVBAMT
E0ppcmkgU2xlemthIHNsZTAwMDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/
VwOD1hlYL6l7GzxNqV1ne7/iMF/gHvPfTwejsC2s9sby7It82qXPRBVA2s1Cjb1A3ucpdlDN
MXM83Lvh881XfkxhS2YLLyiZDmlSzAqfoMLxQ2/E0m1UugttzGJF7/10pEwj0FJFhnIVwA/E
8svCcbhxwO9BBpUz8JG1C6fTd0qyzJtNXVyH+WuHQbU2jgu2JJ7miiEKE1Fis0hFf1rKxTzX
aVGyXiQLOn7TZDfPtXrJEG7eWYlFUP58edyuJELpWHTPHn8xJKYTy8Qq5BgFNyCRQT/6imsh
tZlDBZSEeqyoSNtLsC57ZrjqgtLCEQFK9EX27dOy0/u95zS0OIWdAgMBAAGjggHbMIIB1zAf
BgNVHSMEGDAWgBSMnxEu5uN6BKUeVYtGCASm7ZdwpjAdBgNVHQ4EFgQUF1mSlcyDz9wWit9V
jCz+zJ9CrpswDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUgRJqaXJpLnNsZXprYUBzbHUuY3ow
DgYDVR0PAQH/BAQDAgSwMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDA0BgNVHSAE
LTArMAwGCiqGSIb3TAUCAgEwDAYKYIZIAYb9bAQfATANBgsqhkiG90wFAgMDAzCBhQYDVR0f
BH4wfDA8oDqgOIY2aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL1RFUkVOQWVTY2llbmNlUGVy
c29uYWxDQTMuY3JsMDygOqA4hjZodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vVEVSRU5BZVNj
aWVuY2VQZXJzb25hbENBMy5jcmwwewYIKwYBBQUHAQEEbzBtMCQGCCsGAQUFBzABhhhodHRw
Oi8vb2NzcC5kaWdpY2VydC5jb20wRQYIKwYBBQUHMAKGOWh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9URVJFTkFlU2NpZW5jZVBlcnNvbmFsQ0EzLmNydDANBgkqhkiG9w0BAQsFAAOC
AQEADtFRxKphkcHVdWjR/+i1+cdHfkbicraHlU5Mpw8EX6nemKu4GGAWfzH+Y7p6ImZwUHWf
/SSbrX+57xaFUBOr3jktQm1GRmGUZESEmsUDB8UZXzdQC79/tO9MzRhvEBXuQhdxdoO64Efx
VqtYAB2ydqz7yWh56ioSwaQZEXo5rO1kZuAcmVz8Smd1r/Mur/h8Y+qbrsJng1GS25aMhFts
UV6z9zXuHFkT9Ck8SLdCEDzjzYNjXIDB5n+QOmPXnXrZMlGiI/aOqa5k5Sv6xCIPdH2kbpyd
M1YiH/ChmU9gWJvy0Jq42KGLvWBvuHEzcb3f473Fvn4GWsXu0zDS2oh2/TGCA8MwggO/AgEB
MIGGMHIxCzAJBgNVBAYTAk5MMRYwFAYDVQQIEw1Ob29yZC1Ib2xsYW5kMRIwEAYDVQQHEwlB
bXN0ZXJkYW0xDzANBgNVBAoTBlRFUkVOQTEmMCQGA1UEAxMdVEVSRU5BIGVTY2llbmNlIFBl
cnNvbmFsIENBIDMCEAp5saDxs6+cjJ85YCfhunMwDQYJYIZIAWUDBAIBBQCgggINMBgGCSqG
SIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE3MTEyOTA5MDgyMVowLwYJ
KoZIhvcNAQkEMSIEIFuS5jd9xR3ci09Hj2Qdoq+KmgeiPc6OokdRtOsinA80MGwGCSqGSIb3
DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG
9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZcGCSsG
AQQBgjcQBDGBiTCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDES
MBAGA1UEBxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBl
U2NpZW5jZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMIGZBgsqhkiG9w0BCRAC
CzGBiaCBhjByMQswCQYDVQQGEwJOTDEWMBQGA1UECBMNTm9vcmQtSG9sbGFuZDESMBAGA1UE
BxMJQW1zdGVyZGFtMQ8wDQYDVQQKEwZURVJFTkExJjAkBgNVBAMTHVRFUkVOQSBlU2NpZW5j
ZSBQZXJzb25hbCBDQSAzAhAKebGg8bOvnIyfOWAn4bpzMA0GCSqGSIb3DQEBAQUABIIBAIy4
4CxcJOH1NRuorhCJ7I5HuEeRF31ZM86zderH2J3RnhpAV3uiThbZ+GVwY7To120L87dXElX/
/cdN0apblltVvhIcjASh61dWCgGGSPxGSyfv2P30E3oHsyf8LGMhRinfVbJxbUpy8pUjDwJM
40DkmrSzB1gE9mJKpynqlBrcrZ90eM8DGxZ30Fwh2GikXrd0XYHKcAmNG/NJYHDxosMFyqrK
Odr1e117HUL7BIJHVMakjyXPAfisvyiCMxK9CaNMjkdFY/4bAfSEqIphUPxqONYntgI/zgoR
DVg5i1dh50f9M/KuX1BuAOUlXotIWgc22XNjRzF/7fbWk0Y/l5IAAAAAAAA=
--------------ms050302080405030009040507--
2329
days inactive
2347
days old
13 comments
8 participants
participants (8)
-
Dmitri Chebotarov -
Donny Davis -
Jiří Sléžka -
Kevin Wolf -
Nir Soffer -
Sahina Bose -
Shani Leviim -
Yaniv Kaul