8:36 p.m.
I'm setting up a ovirt cluster using glusterfs and noticing not stellar
performance.
Maybe my setup could use some adjustments?
3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
Each node has 8 spindles configured in 1 array which is split using LVM
with one logical volume for system and one for gluster.
They each have 4 NICs,
NIC1 = ovirtmgmt
NIC2 = gluster
NIC3 = VM traffic
I tried with default glusterfs settings and also with:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB
[root@ovirt3 test scripts]# gluster volume info gv1
Volume Name: gv1
Type: Replicate
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
Options Reconfigured:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB
Using simple dd test on VM in ovirt:
dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
Another VM not in ovirt using nfs:
dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
Is that expected or is there a better way to set it up to get better
performance?
Thanks.
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
3:42 a.m.
+gluster-users
Does disabling 'performance.write-behind' give a better throughput?
On 02/10/2016 11:06 PM, Bill James wrote:
I'm setting up a ovirt cluster using glusterfs and noticing not
stellar performance.
Maybe my setup could use some adjustments?
3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
Each node has 8 spindles configured in 1 array which is split using
LVM with one logical volume for system and one for gluster.
They each have 4 NICs,
NIC1 = ovirtmgmt
NIC2 = gluster
NIC3 = VM traffic
I tried with default glusterfs settings and also with:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB
[root@ovirt3 test scripts]# gluster volume info gv1
Volume Name: gv1
Type: Replicate
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
Options Reconfigured:
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind-window-size: 4MB
Using simple dd test on VM in ovirt:
dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
Another VM not in ovirt using nfs:
dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
Is that expected or is there a better way to set it up to get better
performance?
Thanks.
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2
Global, Inc. and/or its affiliates which may be privileged,
confidential or otherwise protected from disclosure. The information
is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents
of this message is prohibited. If you have received this email in
error please notify the sender by reply e-mail and delete the original
message and any copies. (c) 2015 j2 Global, Inc. All rights reserved.
eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are
registered trademarks of j2 Global, Inc. and its affiliates.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:55 a.m.
On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N <ravishankar(a)redhat.com> wrote:
+gluster-users
Does disabling 'performance.write-behind' give a better throughput?
On 02/10/2016 11:06 PM, Bill James wrote:
>
> I'm setting up a ovirt cluster using glusterfs and noticing not stellar
> performance.
> Maybe my setup could use some adjustments?
>
> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
> Each node has 8 spindles configured in 1 array which is split using LVM
> with one logical volume for system and one for gluster.
> They each have 4 NICs,
> NIC1 = ovirtmgmt
> NIC2 = gluster
How do you ensure that gluster trafic is using this nic?
> NIC3 = VM traffic
How do you ensure that vm trafic is using this nic?
> I tried with default glusterfs settings
And did you find any difference?
> and also with:
> performance.cache-size: 1GB
> performance.readdir-ahead: on
> performance.write-behind-window-size: 4MB
>
> [root@ovirt3 test scripts]# gluster volume info gv1
>
> Volume Name: gv1
> Type: Replicate
> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
> Options Reconfigured:
> performance.cache-size: 1GB
> performance.readdir-ahead: on
> performance.write-behind-window-size: 4MB
>
>
> Using simple dd test on VM in ovirt:
> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
block size of 1G?!
Try 1M (our default for storage operations)
> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>
> Another VM not in ovirt using nfs:
> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>
>
> Is that expected or is there a better way to set it up to get better
> performance?
Adding Niels for advice.
> This email, its contents and attachments contain information from
j2
> Global, Inc. and/or its affiliates which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to be for
> the addressee(s) only. If you are not an addressee, any disclosure, copy,
> distribution, or use of the contents of this message is prohibited. If you
> have received this email in error please notify the sender by reply e-mail
> and delete the original message and any copies. (c) 2015 j2 Global, Inc. All
> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox
> are registered trademarks of j2 Global, Inc. and its affiliates.
Please avoid this, this is a public mailing list, everything you write
here is public.
Nir
9:27 p.m.
thank you for the reply.
We setup gluster using the names associated with NIC 2 IP.
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
That's NIC 2's IP.
Using 'iftop -i eno2 -L 5 -t' :
dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
Peak rate (sent/received/total): 281Mb 5.36Mb
282Mb
Cumulative (sent/received/total): 1.96GB 14.6MB
1.97GB
gluster volume info gv1:
Options Reconfigured:
performance.write-behind-window-size: 4MB
performance.readdir-ahead: on
performance.cache-size: 1GB
performance.write-behind: off
performance.write-behind: off didn't help.
Neither did any other changes I've tried.
There is no VM traffic on this VM right now except my test.
On 02/10/2016 11:55 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
<ravishankar(a)redhat.com> wrote:
> +gluster-users
>
> Does disabling 'performance.write-behind' give a better throughput?
>
>
>
> On 02/10/2016 11:06 PM, Bill James wrote:
>> I'm setting up a ovirt cluster using glusterfs and noticing not stellar
>> performance.
>> Maybe my setup could use some adjustments?
>>
>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
>> Each node has 8 spindles configured in 1 array which is split using LVM
>> with one logical volume for system and one for gluster.
>> They each have 4 NICs,
>> NIC1 = ovirtmgmt
>> NIC2 = gluster (1GbE)
How do you ensure that gluster trafic is using this nic?
>> NIC3 = VM traffic
How do you ensure that vm trafic is using this nic?
>> I tried with default glusterfs settings
And did you find any difference?
>> and also with:
>> performance.cache-size: 1GB
>> performance.readdir-ahead: on
>> performance.write-behind-window-size: 4MB
>>
>> [root@ovirt3 test scripts]# gluster volume info gv1
>>
>> Volume Name: gv1
>> Type: Replicate
>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>> Status: Started
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>> Options Reconfigured:
>> performance.cache-size: 1GB
>> performance.readdir-ahead: on
>> performance.write-behind-window-size: 4MB
>>
>>
>> Using simple dd test on VM in ovirt:
>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
block size of 1G?!
Try 1M (our default for storage operations)
>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>>
>> Another VM not in ovirt using nfs:
>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>>
>>
>> Is that expected or is there a better way to set it up to get better
>> performance?
Adding Niels for advice.
>> This email, its contents and ....
Please avoid this, this is a public mailing list, everything you write
here is public.
Nir
I'll have to look into how to remove this sig for this mailing list....
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
11:28 p.m.
On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com> wrote:
thank you for the reply.
We setup gluster using the names associated with NIC 2 IP.
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
That's NIC 2's IP.
Using 'iftop -i eno2 -L 5 -t' :
dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
Can you share the xml of this vm? You can find it in vdsm log,
at the time you start the vm.
Or you can do (on the host):
# virsh
virsh # list
(username: vdsm@ovirt password: shibboleth)
virsh # dumpxml vm-id
Peak rate (sent/received/total): 281Mb 5.36Mb
282Mb
Cumulative (sent/received/total): 1.96GB 14.6MB
1.97GB
gluster volume info gv1:
Options Reconfigured:
performance.write-behind-window-size: 4MB
performance.readdir-ahead: on
performance.cache-size: 1GB
performance.write-behind: off
performance.write-behind: off didn't help.
Neither did any other changes I've tried.
There is no VM traffic on this VM right now except my test.
On 02/10/2016 11:55 PM, Nir Soffer wrote:
>
> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N <ravishankar(a)redhat.com>
> wrote:
>>
>> +gluster-users
>>
>> Does disabling 'performance.write-behind' give a better throughput?
>>
>>
>>
>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>
>>> I'm setting up a ovirt cluster using glusterfs and noticing not stellar
>>> performance.
>>> Maybe my setup could use some adjustments?
>>>
>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
>>> Each node has 8 spindles configured in 1 array which is split using LVM
>>> with one logical volume for system and one for gluster.
>>> They each have 4 NICs,
>>> NIC1 = ovirtmgmt
>>> NIC2 = gluster (1GbE)
>
> How do you ensure that gluster trafic is using this nic?
>
>>> NIC3 = VM traffic
>
> How do you ensure that vm trafic is using this nic?
>
>>> I tried with default glusterfs settings
>
> And did you find any difference?
>
>>> and also with:
>>> performance.cache-size: 1GB
>>> performance.readdir-ahead: on
>>> performance.write-behind-window-size: 4MB
>>>
>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>
>>> Volume Name: gv1
>>> Type: Replicate
>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>> Status: Started
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>> Options Reconfigured:
>>> performance.cache-size: 1GB
>>> performance.readdir-ahead: on
>>> performance.write-behind-window-size: 4MB
>>>
>>>
>>> Using simple dd test on VM in ovirt:
>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>
> block size of 1G?!
>
> Try 1M (our default for storage operations)
>
>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>>>
>>> Another VM not in ovirt using nfs:
>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>>>
>>>
>>> Is that expected or is there a better way to set it up to get better
>>> performance?
>
> Adding Niels for advice.
>
>>> This email, its contents and ....
>
> Please avoid this, this is a public mailing list, everything you write
> here is public.
>
> Nir
I'll have to look into how to remove this sig for this mailing list....
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global,
Inc. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for
the addressee(s) only. If you are not an addressee, any disclosure, copy,
distribution, or use of the contents of this message is prohibited. If you
have received this email in error please notify the sender by reply e-mail
and delete the original message and any copies. (c) 2015 j2 Global, Inc. All
rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox
are registered trademarks of j2 Global, Inc. and its affiliates.
2:13 a.m.
xml attached.
On 02/11/2016 12:28 PM, Nir Soffer wrote:
On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com>
wrote:
> thank you for the reply.
>
> We setup gluster using the names associated with NIC 2 IP.
> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>
> That's NIC 2's IP.
> Using 'iftop -i eno2 -L 5 -t' :
>
> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
Can you share the xml of this vm? You can find it in vdsm log,
at the time you start the vm.
Or you can do (on the host):
# virsh
virsh # list
(username: vdsm@ovirt password: shibboleth)
virsh # dumpxml vm-id
> Peak rate (sent/received/total): 281Mb 5.36Mb
> 282Mb
> Cumulative (sent/received/total): 1.96GB 14.6MB
> 1.97GB
>
> gluster volume info gv1:
> Options Reconfigured:
> performance.write-behind-window-size: 4MB
> performance.readdir-ahead: on
> performance.cache-size: 1GB
> performance.write-behind: off
>
> performance.write-behind: off didn't help.
> Neither did any other changes I've tried.
>
>
> There is no VM traffic on this VM right now except my test.
>
>
>
> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N <ravishankar(a)redhat.com>
>> wrote:
>>> +gluster-users
>>>
>>> Does disabling 'performance.write-behind' give a better throughput?
>>>
>>>
>>>
>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>> I'm setting up a ovirt cluster using glusterfs and noticing not
stellar
>>>> performance.
>>>> Maybe my setup could use some adjustments?
>>>>
>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
>>>> Each node has 8 spindles configured in 1 array which is split using LVM
>>>> with one logical volume for system and one for gluster.
>>>> They each have 4 NICs,
>>>> NIC1 = ovirtmgmt
>>>> NIC2 = gluster (1GbE)
>> How do you ensure that gluster trafic is using this nic?
>>
>>>> NIC3 = VM traffic
>> How do you ensure that vm trafic is using this nic?
>>
>>>> I tried with default glusterfs settings
>> And did you find any difference?
>>
>>>> and also with:
>>>> performance.cache-size: 1GB
>>>> performance.readdir-ahead: on
>>>> performance.write-behind-window-size: 4MB
>>>>
>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>
>>>> Volume Name: gv1
>>>> Type: Replicate
>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>> Status: Started
>>>> Number of Bricks: 1 x 3 = 3
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>> Options Reconfigured:
>>>> performance.cache-size: 1GB
>>>> performance.readdir-ahead: on
>>>> performance.write-behind-window-size: 4MB
>>>>
>>>>
>>>> Using simple dd test on VM in ovirt:
>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>> block size of 1G?!
>>
>> Try 1M (our default for storage operations)
>>
>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>>>>
>>>> Another VM not in ovirt using nfs:
>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>>>>
>>>>
>>>> Is that expected or is there a better way to set it up to get better
>>>> performance?
>> Adding Niels for advice.
>>
>>>> This email, its contents and ....
>> Please avoid this, this is a public mailing list, everything you write
>> here is public.
>>
>> Nir
> I'll have to look into how to remove this sig for this mailing list....
>
> Cloud Services for Business www.j2.com
> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>
>
> This email, its contents and attachments contain information from j2 Global,
> Inc. and/or its affiliates which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to be for
> the addressee(s) only. If you are not an addressee, any disclosure, copy,
> distribution, or use of the contents of this message is prohibited. If you
> have received this email in error please notify the sender by reply e-mail
> and delete the original message and any copies. (c) 2015 j2 Global, Inc. All
> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox
> are registered trademarks of j2 Global, Inc. and its affiliates.
2:27 a.m.
--------------000400070400010206070402
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
don't know if it helps, but I ran a few more tests, all from the same
hardware node.
The VM:
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
Writing directly to gluster volume:
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
Writing to NFS volume:
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/storage/qa/testfile
bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
NFS & Gluster are using the same interface. Tests were not run at same time.
This would suggest my problem isn't glusterfs, but the VM performance.
On 02/11/2016 03:13 PM, Bill James wrote:
xml attached.
On 02/11/2016 12:28 PM, Nir Soffer wrote:
> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com> wrote:
>> thank you for the reply.
>>
>> We setup gluster using the names associated with NIC 2 IP.
>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>
>> That's NIC 2's IP.
>> Using 'iftop -i eno2 -L 5 -t' :
>>
>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
> Can you share the xml of this vm? You can find it in vdsm log,
> at the time you start the vm.
>
> Or you can do (on the host):
>
> # virsh
> virsh # list
> (username: vdsm@ovirt password: shibboleth)
> virsh # dumpxml vm-id
>
>> Peak rate (sent/received/total): 281Mb 5.36Mb
>> 282Mb
>> Cumulative (sent/received/total): 1.96GB 14.6MB
>> 1.97GB
>>
>> gluster volume info gv1:
>> Options Reconfigured:
>> performance.write-behind-window-size: 4MB
>> performance.readdir-ahead: on
>> performance.cache-size: 1GB
>> performance.write-behind: off
>>
>> performance.write-behind: off didn't help.
>> Neither did any other changes I've tried.
>>
>>
>> There is no VM traffic on this VM right now except my test.
>>
>>
>>
>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
>>> <ravishankar(a)redhat.com>
>>> wrote:
>>>> +gluster-users
>>>>
>>>> Does disabling 'performance.write-behind' give a better
throughput?
>>>>
>>>>
>>>>
>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>> I'm setting up a ovirt cluster using glusterfs and noticing not
>>>>> stellar
>>>>> performance.
>>>>> Maybe my setup could use some adjustments?
>>>>>
>>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt
>>>>> 3.6.2.6-1.
>>>>> Each node has 8 spindles configured in 1 array which is split
>>>>> using LVM
>>>>> with one logical volume for system and one for gluster.
>>>>> They each have 4 NICs,
>>>>> NIC1 = ovirtmgmt
>>>>> NIC2 = gluster (1GbE)
>>> How do you ensure that gluster trafic is using this nic?
>>>
>>>>> NIC3 = VM traffic
>>> How do you ensure that vm trafic is using this nic?
>>>
>>>>> I tried with default glusterfs settings
>>> And did you find any difference?
>>>
>>>>> and also with:
>>>>> performance.cache-size: 1GB
>>>>> performance.readdir-ahead: on
>>>>> performance.write-behind-window-size: 4MB
>>>>>
>>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>>
>>>>> Volume Name: gv1
>>>>> Type: Replicate
>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>> Status: Started
>>>>> Number of Bricks: 1 x 3 = 3
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Options Reconfigured:
>>>>> performance.cache-size: 1GB
>>>>> performance.readdir-ahead: on
>>>>> performance.write-behind-window-size: 4MB
>>>>>
>>>>>
>>>>> Using simple dd test on VM in ovirt:
>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>> block size of 1G?!
>>>
>>> Try 1M (our default for storage operations)
>>>
>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>>>>>
>>>>> Another VM not in ovirt using nfs:
>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>>>>>
>>>>>
>>>>> Is that expected or is there a better way to set it up to get better
>>>>> performance?
>>> Adding Niels for advice.
>>>
>>>>> This email, its contents and ....
>>> Please avoid this, this is a public mailing list, everything you write
>>> here is public.
>>>
>>> Nir
>> I'll have to look into how to remove this sig for this mailing list....
>>
>> Cloud Services for Business www.j2.com
>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>
>>
>> This email, its contents and attachments contain information from j2
>> Global,
>> Inc. and/or its affiliates which may be privileged, confidential or
>> otherwise protected from disclosure. The information is intended to
>> be for
>> the addressee(s) only. If you are not an addressee, any disclosure,
>> copy,
>> distribution, or use of the contents of this message is prohibited.
>> If you
>> have received this email in error please notify the sender by reply
>> e-mail
>> and delete the original message and any copies. (c) 2015 j2 Global,
>> Inc. All
>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and
>> Onebox
>> are registered trademarks of j2 Global, Inc. and its affiliates.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
--------------000400070400010206070402
Content-Type: text/html; charset="windows-1252"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
don't know if it helps, but I ran a few more tests, all from the
same hardware node.<br>
<br>
The VM:<br>
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s<br>
<br>
Writing directly to gluster volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s<br>
<br>
<br>
Writing to NFS volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s<br>
<br>
NFS & Gluster are using the same interface. Tests were not run
at same time.<br>
<br>
This would suggest my problem isn't glusterfs, but the VM
performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:13 PM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:56BD1589.8090506@j2.com"
type="cite">xml
attached.
<br>
<br>
<br>
On 02/11/2016 12:28 PM, Nir Soffer wrote:
<br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 8:27 PM, Bill
James <a class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote:
<br>
<blockquote type="cite">thank you for the reply.
<br>
<br>
We setup gluster using the names associated with NIC 2 IP.
<br>
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
<br>
That's NIC 2's IP.
<br>
Using 'iftop -i eno2 -L 5 -t' :
<br>
<br>
dd if=/dev/zero of=/root/testfile bs=1M count=1000
oflag=direct
<br>
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
<br>
</blockquote>
Can you share the xml of this vm? You can find it in vdsm log,
<br>
at the time you start the vm.
<br>
<br>
Or you can do (on the host):
<br>
<br>
# virsh
<br>
virsh # list
<br>
(username: vdsm@ovirt password: shibboleth)
<br>
virsh # dumpxml vm-id
<br>
<br>
<blockquote type="cite">Peak rate
(sent/received/total): 281Mb 5.36Mb
<br>
282Mb
<br>
Cumulative (sent/received/total): 1.96GB
14.6MB
<br>
1.97GB
<br>
<br>
gluster volume info gv1:
<br>
Options Reconfigured:
<br>
performance.write-behind-window-size: 4MB
<br>
performance.readdir-ahead: on
<br>
performance.cache-size: 1GB
<br>
performance.write-behind: off
<br>
<br>
performance.write-behind: off didn't help.
<br>
Neither did any other changes I've tried.
<br>
<br>
<br>
There is no VM traffic on this VM right now except my test.
<br>
<br>
<br>
<br>
On 02/10/2016 11:55 PM, Nir Soffer wrote:
<br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 2:42 AM,
Ravishankar N <a class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a>
<br>
wrote:
<br>
<blockquote type="cite">+gluster-users
<br>
<br>
Does disabling 'performance.write-behind' give a better
throughput?
<br>
<br>
<br>
<br>
On 02/10/2016 11:06 PM, Bill James wrote:
<br>
<blockquote type="cite">I'm setting up a ovirt cluster
using glusterfs and noticing not stellar
<br>
performance.
<br>
Maybe my setup could use some adjustments?
<br>
<br>
3 hardware nodes running centos7.2, glusterfs 3.7.6.1,
ovirt 3.6.2.6-1.
<br>
Each node has 8 spindles configured in 1 array which is
split using LVM
<br>
with one logical volume for system and one for gluster.
<br>
They each have 4 NICs,
<br>
NIC1 = ovirtmgmt
<br>
NIC2 = gluster (1GbE)
<br>
</blockquote>
</blockquote>
How do you ensure that gluster trafic is using this nic?
<br>
<br>
<blockquote type="cite">
<blockquote type="cite"> NIC3 = VM traffic
<br>
</blockquote>
</blockquote>
How do you ensure that vm trafic is using this nic?
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">I tried with default glusterfs
settings
<br>
</blockquote>
</blockquote>
And did you find any difference?
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">and also with:
<br>
performance.cache-size: 1GB
<br>
performance.readdir-ahead: on
<br>
performance.write-behind-window-size: 4MB
<br>
<br>
[root@ovirt3 test scripts]# gluster volume info gv1
<br>
<br>
Volume Name: gv1
<br>
Type: Replicate
<br>
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
<br>
Status: Started
<br>
Number of Bricks: 1 x 3 = 3
<br>
Transport-type: tcp
<br>
Bricks:
<br>
Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Options Reconfigured:
<br>
performance.cache-size: 1GB
<br>
performance.readdir-ahead: on
<br>
performance.write-behind-window-size: 4MB
<br>
<br>
<br>
Using simple dd test on VM in ovirt:
<br>
dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
<br>
</blockquote>
</blockquote>
block size of 1G?!
<br>
<br>
Try 1M (our default for storage operations)
<br>
<br>
<blockquote type="cite">
<blockquote type="cite"> 1073741824 bytes (1.1 GB)
copied, 65.9337 s, 16.3 MB/s
<br>
<br>
Another VM not in ovirt using nfs:
<br>
dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
<br>
1073741824 bytes (1.1 GB) copied, 27.0079 s,
39.8 MB/s
<br>
<br>
<br>
Is that expected or is there a better way to set it up
to get better
<br>
performance?
<br>
</blockquote>
</blockquote>
Adding Niels for advice.
<br>
<br>
<blockquote type="cite">
<blockquote type="cite">This email, its contents and ....
<br>
</blockquote>
</blockquote>
Please avoid this, this is a public mailing list, everything
you write
<br>
here is public.
<br>
<br>
Nir
<br>
</blockquote>
I'll have to look into how to remove this sig for this mailing
list....
<br>
<br>
Cloud Services for Business <a class="moz-txt-link-abbreviated"
href="http://www.j2.com">www.j2.com</a>
<br>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe |
Onebox
<br>
<br>
<br>
This email, its contents and attachments contain information
from j2 Global,
<br>
Inc. and/or its affiliates which may be privileged,
confidential or
<br>
otherwise protected from disclosure. The information is
intended to be for
<br>
the addressee(s) only. If you are not an addressee, any
disclosure, copy,
<br>
distribution, or use of the contents of this message is
prohibited. If you
<br>
have received this email in error please notify the sender by
reply e-mail
<br>
and delete the original message and any copies. (c) 2015 j2
Global, Inc. All
<br>
rights reserved. eFax, eVoice, Campaigner, FuseMail,
KeepItSafe, and Onebox
<br>
are registered trademarks of j2 Global, Inc. and its
affiliates.
<br>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
style='color:windowtext;
text-decoration:none'><img border=0 width=391 height=46
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com"></span></a></p>
<p><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:gray'>This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p></body>
</html>
--------------000400070400010206070402--
![src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...](http://src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Footer.jpg%22){kind=link}
2:52 a.m.
--------------040903020404050203020000
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 8bit
My apologies, I'm showing how much of a noob I am.
Ignore last direct to gluster numbers, as that wasn't really glusterfs.
[root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com:/gv1
/mnt/tmp/
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 bs=1M
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
That's more how I expected, it is pointing to glusterfs performance.
On 02/11/2016 03:27 PM, Bill James wrote:
don't know if it helps, but I ran a few more tests, all from the
same
hardware node.
The VM:
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
Writing directly to gluster volume:
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
Writing to NFS volume:
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/storage/qa/testfile
bs=1M count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
NFS & Gluster are using the same interface. Tests were not run at same
time.
This would suggest my problem isn't glusterfs, but the VM performance.
On 02/11/2016 03:13 PM, Bill James wrote:
> xml attached.
>
>
> On 02/11/2016 12:28 PM, Nir Soffer wrote:
>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com> wrote:
>>> thank you for the reply.
>>>
>>> We setup gluster using the names associated with NIC 2 IP.
>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>
>>> That's NIC 2's IP.
>>> Using 'iftop -i eno2 -L 5 -t' :
>>>
>>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
>> Can you share the xml of this vm? You can find it in vdsm log,
>> at the time you start the vm.
>>
>> Or you can do (on the host):
>>
>> # virsh
>> virsh # list
>> (username: vdsm@ovirt password: shibboleth)
>> virsh # dumpxml vm-id
>>
>>> Peak rate (sent/received/total): 281Mb 5.36Mb
>>> 282Mb
>>> Cumulative (sent/received/total): 1.96GB 14.6MB
>>> 1.97GB
>>>
>>> gluster volume info gv1:
>>> Options Reconfigured:
>>> performance.write-behind-window-size: 4MB
>>> performance.readdir-ahead: on
>>> performance.cache-size: 1GB
>>> performance.write-behind: off
>>>
>>> performance.write-behind: off didn't help.
>>> Neither did any other changes I've tried.
>>>
>>>
>>> There is no VM traffic on this VM right now except my test.
>>>
>>>
>>>
>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
>>>> <ravishankar(a)redhat.com>
>>>> wrote:
>>>>> +gluster-users
>>>>>
>>>>> Does disabling 'performance.write-behind' give a better
throughput?
>>>>>
>>>>>
>>>>>
>>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>>> I'm setting up a ovirt cluster using glusterfs and noticing
not
>>>>>> stellar
>>>>>> performance.
>>>>>> Maybe my setup could use some adjustments?
>>>>>>
>>>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt
>>>>>> 3.6.2.6-1.
>>>>>> Each node has 8 spindles configured in 1 array which is split
>>>>>> using LVM
>>>>>> with one logical volume for system and one for gluster.
>>>>>> They each have 4 NICs,
>>>>>> NIC1 = ovirtmgmt
>>>>>> NIC2 = gluster (1GbE)
>>>> How do you ensure that gluster trafic is using this nic?
>>>>
>>>>>> NIC3 = VM traffic
>>>> How do you ensure that vm trafic is using this nic?
>>>>
>>>>>> I tried with default glusterfs settings
>>>> And did you find any difference?
>>>>
>>>>>> and also with:
>>>>>> performance.cache-size: 1GB
>>>>>> performance.readdir-ahead: on
>>>>>> performance.write-behind-window-size: 4MB
>>>>>>
>>>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>>>
>>>>>> Volume Name: gv1
>>>>>> Type: Replicate
>>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>>> Status: Started
>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>> Transport-type: tcp
>>>>>> Bricks:
>>>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>> Options Reconfigured:
>>>>>> performance.cache-size: 1GB
>>>>>> performance.readdir-ahead: on
>>>>>> performance.write-behind-window-size: 4MB
>>>>>>
>>>>>>
>>>>>> Using simple dd test on VM in ovirt:
>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>>> block size of 1G?!
>>>>
>>>> Try 1M (our default for storage operations)
>>>>
>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>>>>>>
>>>>>> Another VM not in ovirt using nfs:
>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>>>>>>
>>>>>>
>>>>>> Is that expected or is there a better way to set it up to get
>>>>>> better
>>>>>> performance?
>>>> Adding Niels for advice.
>>>>
>>>>>> This email, its contents and ....
>>>> Please avoid this, this is a public mailing list, everything you
>>>> write
>>>> here is public.
>>>>
>>>> Nir
>>> I'll have to look into how to remove this sig for this mailing
>>> list....
>>>
>>> Cloud Services for Business www.j2.com
>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>>
>>>
>>> This email, its contents and attachments contain information from
>>> j2 Global,
>>> Inc. and/or its affiliates which may be privileged, confidential or
>>> otherwise protected from disclosure. The information is intended to
>>> be for
>>> the addressee(s) only. If you are not an addressee, any disclosure,
>>> copy,
>>> distribution, or use of the contents of this message is prohibited.
>>> If you
>>> have received this email in error please notify the sender by reply
>>> e-mail
>>> and delete the original message and any copies. (c) 2015 j2 Global,
>>> Inc. All
>>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe,
>>> and Onebox
>>> are registered trademarks of j2 Global, Inc. and its affiliates.
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
www.j2.com
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
This email, its contents and attachments contain information from j2
Global, Inc
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be
for the addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of this message
is prohibited. If you have received this email in error please notify
the sender by reply e-mail and delete the original message and any
copies. © 2015 j2 Global, Inc <http://www.j2.com/>;. All rights
reserved. eFax ® <http://www.efax.com/>;, eVoice ®
<http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
<http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/> are
r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. and its
affiliates.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
--------------040903020404050203020000
Content-Type: text/html; charset="windows-1252"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
My apologies, I'm showing how much of a noob I am.<br>
Ignore last direct to gluster numbers, as that wasn't really
glusterfs.<br>
<br>
<br>
[root@ovirt2 test ~]# mount -t glusterfs
ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/<br>
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s<br>
<br>
That's more how I expected, it is pointing to glusterfs performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:27 PM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:56BD18D7.8080909@j2.com" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
don't know if it helps, but I ran a few more tests, all from the
same hardware node.<br>
<br>
The VM:<br>
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s<br>
<br>
Writing directly to gluster volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000
oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s<br>
<br>
<br>
Writing to NFS volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s<br>
<br>
NFS & Gluster are using the same interface. Tests were not run
at same time.<br>
<br>
This would suggest my problem isn't glusterfs, but the VM
performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:13 PM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:56BD1589.8090506@j2.com"
type="cite">xml
attached. <br>
<br>
<br>
On 02/11/2016 12:28 PM, Nir Soffer wrote: <br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 8:27 PM, Bill
James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote: <br>
<blockquote type="cite">thank you for the reply. <br>
<br>
We setup gluster using the names associated with NIC 2 IP.
<br>
Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
<br>
That's NIC 2's IP. <br>
Using 'iftop -i eno2 -L 5 -t' : <br>
<br>
dd if=/dev/zero of=/root/testfile bs=1M count=1000
oflag=direct <br>
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s <br>
</blockquote>
Can you share the xml of this vm? You can find it in vdsm log,
<br>
at the time you start the vm. <br>
<br>
Or you can do (on the host): <br>
<br>
# virsh <br>
virsh # list <br>
(username: vdsm@ovirt password: shibboleth) <br>
virsh # dumpxml vm-id <br>
<br>
<blockquote type="cite">Peak rate
(sent/received/total): 281Mb 5.36Mb <br>
282Mb <br>
Cumulative (sent/received/total): 1.96GB
14.6MB <br>
1.97GB <br>
<br>
gluster volume info gv1: <br>
Options Reconfigured: <br>
performance.write-behind-window-size: 4MB <br>
performance.readdir-ahead: on <br>
performance.cache-size: 1GB <br>
performance.write-behind: off <br>
<br>
performance.write-behind: off didn't help. <br>
Neither did any other changes I've tried. <br>
<br>
<br>
There is no VM traffic on this VM right now except my test.
<br>
<br>
<br>
<br>
On 02/10/2016 11:55 PM, Nir Soffer wrote: <br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 2:42 AM,
Ravishankar N <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a>
<br>
wrote: <br>
<blockquote type="cite">+gluster-users <br>
<br>
Does disabling 'performance.write-behind' give a better
throughput? <br>
<br>
<br>
<br>
On 02/10/2016 11:06 PM, Bill James wrote: <br>
<blockquote type="cite">I'm setting up a ovirt
cluster
using glusterfs and noticing not stellar <br>
performance. <br>
Maybe my setup could use some adjustments? <br>
<br>
3 hardware nodes running centos7.2, glusterfs 3.7.6.1,
ovirt 3.6.2.6-1. <br>
Each node has 8 spindles configured in 1 array which
is split using LVM <br>
with one logical volume for system and one for
gluster. <br>
They each have 4 NICs, <br>
NIC1 = ovirtmgmt <br>
NIC2 = gluster (1GbE) <br>
</blockquote>
</blockquote>
How do you ensure that gluster trafic is using this nic? <br>
<br>
<blockquote type="cite">
<blockquote type="cite"> NIC3 = VM traffic <br>
</blockquote>
</blockquote>
How do you ensure that vm trafic is using this nic? <br>
<br>
<blockquote type="cite">
<blockquote type="cite">I tried with default glusterfs
settings <br>
</blockquote>
</blockquote>
And did you find any difference? <br>
<br>
<blockquote type="cite">
<blockquote type="cite">and also with: <br>
performance.cache-size: 1GB <br>
performance.readdir-ahead: on <br>
performance.write-behind-window-size: 4MB <br>
<br>
[root@ovirt3 test scripts]# gluster volume info gv1 <br>
<br>
Volume Name: gv1 <br>
Type: Replicate <br>
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 <br>
Status: Started <br>
Number of Bricks: 1 x 3 = 3 <br>
Transport-type: tcp <br>
Bricks: <br>
Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Options Reconfigured: <br>
performance.cache-size: 1GB <br>
performance.readdir-ahead: on <br>
performance.write-behind-window-size: 4MB <br>
<br>
<br>
Using simple dd test on VM in ovirt: <br>
dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct <br>
</blockquote>
</blockquote>
block size of 1G?! <br>
<br>
Try 1M (our default for storage operations) <br>
<br>
<blockquote type="cite">
<blockquote type="cite"> 1073741824 bytes (1.1
GB) copied, 65.9337 s, 16.3 MB/s <br>
<br>
Another VM not in ovirt using nfs: <br>
dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct <br>
1073741824 bytes (1.1 GB) copied, 27.0079 s,
39.8 MB/s <br>
<br>
<br>
Is that expected or is there a better way to set it up
to get better <br>
performance? <br>
</blockquote>
</blockquote>
Adding Niels for advice. <br>
<br>
<blockquote type="cite">
<blockquote type="cite">This email, its contents and
.... <br>
</blockquote>
</blockquote>
Please avoid this, this is a public mailing list,
everything you write <br>
here is public. <br>
<br>
Nir <br>
</blockquote>
I'll have to look into how to remove this sig for this
mailing list.... <br>
<br>
Cloud Services for Business <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="http://www.j2.com">www.j2.com</a>
<br>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe |
Onebox <br>
<br>
<br>
This email, its contents and attachments contain information
from j2 Global, <br>
Inc. and/or its affiliates which may be privileged,
confidential or <br>
otherwise protected from disclosure. The information is
intended to be for <br>
the addressee(s) only. If you are not an addressee, any
disclosure, copy, <br>
distribution, or use of the contents of this message is
prohibited. If you <br>
have received this email in error please notify the sender
by reply e-mail <br>
and delete the original message and any copies. (c) 2015 j2
Global, Inc. All <br>
rights reserved. eFax, eVoice, Campaigner, FuseMail,
KeepItSafe, and Onebox <br>
are registered trademarks of j2 Global, Inc. and its
affiliates. <br>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
style="color:windowtext;
text-decoration:none"><img moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" border="0" height="46"
width="391"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of this
message is prohibited. If you have received this email in
error please notify the sender by reply e-mail and delete the
original message and any copies. © 2015 <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global,
Inc</a>. All rights reserved. <a moz-do-not-send="true"
href="http://www.efax.com/">eFax ®</a>, <a
moz-do-not-send="true"
href="http://www.evoice.com/">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner ®</a>, <a
moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and
<a
moz-do-not-send="true"
href="http://www.onebox.com/">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global,
Inc</a>. and its affiliates.</span></p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
style='color:windowtext;
text-decoration:none'><img border=0 width=391 height=46
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com"></span></a></p>
<p><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:gray'>This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p></body>
</html>
--------------040903020404050203020000--
7:18 a.m.
This is a multi-part message in MIME format.
--------------020701000405040103030307
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
Hi Bill,
Can you enable virt-profile setting for your volume and see if that
helps? You need to enable this optimization when you create the volume
using ovrit, or use the following command for an existing volume:
#gluster volume set <volname> group virt
-Ravi
On 02/12/2016 05:22 AM, Bill James wrote:
My apologies, I'm showing how much of a noob I am.
Ignore last direct to gluster numbers, as that wasn't really glusterfs.
[root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com:/gv1
/mnt/tmp/
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 bs=1M
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
That's more how I expected, it is pointing to glusterfs performance.
On 02/11/2016 03:27 PM, Bill James wrote:
> don't know if it helps, but I ran a few more tests, all from the same
> hardware node.
>
> The VM:
> [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
> count=1000 oflag=direct
> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
>
> Writing directly to gluster volume:
> [root@ovirt2 test ~]# time dd if=/dev/zero
> of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
>
>
> Writing to NFS volume:
> [root@ovirt2 test ~]# time dd if=/dev/zero
> of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct
> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
>
> NFS & Gluster are using the same interface. Tests were not run at
> same time.
>
> This would suggest my problem isn't glusterfs, but the VM performance.
>
>
>
> On 02/11/2016 03:13 PM, Bill James wrote:
>> xml attached.
>>
>>
>> On 02/11/2016 12:28 PM, Nir Soffer wrote:
>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com> wrote:
>>>> thank you for the reply.
>>>>
>>>> We setup gluster using the names associated with NIC 2 IP.
>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>
>>>> That's NIC 2's IP.
>>>> Using 'iftop -i eno2 -L 5 -t' :
>>>>
>>>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
>>> Can you share the xml of this vm? You can find it in vdsm log,
>>> at the time you start the vm.
>>>
>>> Or you can do (on the host):
>>>
>>> # virsh
>>> virsh # list
>>> (username: vdsm@ovirt password: shibboleth)
>>> virsh # dumpxml vm-id
>>>
>>>> Peak rate (sent/received/total): 281Mb 5.36Mb
>>>> 282Mb
>>>> Cumulative (sent/received/total): 1.96GB 14.6MB
>>>> 1.97GB
>>>>
>>>> gluster volume info gv1:
>>>> Options Reconfigured:
>>>> performance.write-behind-window-size: 4MB
>>>> performance.readdir-ahead: on
>>>> performance.cache-size: 1GB
>>>> performance.write-behind: off
>>>>
>>>> performance.write-behind: off didn't help.
>>>> Neither did any other changes I've tried.
>>>>
>>>>
>>>> There is no VM traffic on this VM right now except my test.
>>>>
>>>>
>>>>
>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
>>>>> <ravishankar(a)redhat.com>
>>>>> wrote:
>>>>>> +gluster-users
>>>>>>
>>>>>> Does disabling 'performance.write-behind' give a better
throughput?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>>>> I'm setting up a ovirt cluster using glusterfs and
noticing not
>>>>>>> stellar
>>>>>>> performance.
>>>>>>> Maybe my setup could use some adjustments?
>>>>>>>
>>>>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1, ovirt
>>>>>>> 3.6.2.6-1.
>>>>>>> Each node has 8 spindles configured in 1 array which is split
>>>>>>> using LVM
>>>>>>> with one logical volume for system and one for gluster.
>>>>>>> They each have 4 NICs,
>>>>>>> NIC1 = ovirtmgmt
>>>>>>> NIC2 = gluster (1GbE)
>>>>> How do you ensure that gluster trafic is using this nic?
>>>>>
>>>>>>> NIC3 = VM traffic
>>>>> How do you ensure that vm trafic is using this nic?
>>>>>
>>>>>>> I tried with default glusterfs settings
>>>>> And did you find any difference?
>>>>>
>>>>>>> and also with:
>>>>>>> performance.cache-size: 1GB
>>>>>>> performance.readdir-ahead: on
>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>
>>>>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>>>>
>>>>>>> Volume Name: gv1
>>>>>>> Type: Replicate
>>>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>>>> Status: Started
>>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>> Options Reconfigured:
>>>>>>> performance.cache-size: 1GB
>>>>>>> performance.readdir-ahead: on
>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>
>>>>>>>
>>>>>>> Using simple dd test on VM in ovirt:
>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>> block size of 1G?!
>>>>>
>>>>> Try 1M (our default for storage operations)
>>>>>
>>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3
MB/s
>>>>>>>
>>>>>>> Another VM not in ovirt using nfs:
>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8
MB/s
>>>>>>>
>>>>>>>
>>>>>>> Is that expected or is there a better way to set it up to get
>>>>>>> better
>>>>>>> performance?
>>>>> Adding Niels for advice.
>>>>>
>>>>>>> This email, its contents and ....
>>>>> Please avoid this, this is a public mailing list, everything you
>>>>> write
>>>>> here is public.
>>>>>
>>>>> Nir
>>>> I'll have to look into how to remove this sig for this mailing
>>>> list....
>>>>
>>>> Cloud Services for Business www.j2.com
>>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>>>
>>>>
>>>> This email, its contents and attachments contain information from
>>>> j2 Global,
>>>> Inc. and/or its affiliates which may be privileged, confidential or
>>>> otherwise protected from disclosure. The information is intended
>>>> to be for
>>>> the addressee(s) only. If you are not an addressee, any
>>>> disclosure, copy,
>>>> distribution, or use of the contents of this message is
>>>> prohibited. If you
>>>> have received this email in error please notify the sender by
>>>> reply e-mail
>>>> and delete the original message and any copies. (c) 2015 j2
>>>> Global, Inc. All
>>>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe,
>>>> and Onebox
>>>> are registered trademarks of j2 Global, Inc. and its affiliates.
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> www.j2.com
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>
> This email, its contents and attachments contain information from j2
> Global, Inc
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
> and/or its affiliates which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to
> be for the addressee(s) only. If you are not an addressee, any
> disclosure, copy, distribution, or use of the contents of this
> message is prohibited. If you have received this email in error
> please notify the sender by reply e-mail and delete the original
> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;.
> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ®
> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/>
> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
> and its affiliates.
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
www.j2.com
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
This email, its contents and attachments contain information from j2
Global, Inc
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be
for the addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of this message
is prohibited. If you have received this email in error please notify
the sender by reply e-mail and delete the original message and any
copies. © 2015 j2 Global, Inc <http://www.j2.com/>;. All rights
reserved. eFax ® <http://www.efax.com/>;, eVoice ®
<http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
<http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/> are
r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. and its
affiliates.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------020701000405040103030307
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Bill,<br>
Can you enable virt-profile setting for your volume and see if
that helps? You need to enable this optimization when you create
the volume using ovrit, or use the following command for an
existing volume:<br>
<tt><br>
#gluster volume set <volname> group virt</tt><br>
<br>
-Ravi<br>
<br>
<br>
On 02/12/2016 05:22 AM, Bill James wrote:<br>
</div>
<blockquote cite="mid:56BD1EB9.5000802@j2.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
My apologies, I'm showing how much of a noob I am.<br>
Ignore last direct to gluster numbers, as that wasn't really
glusterfs.<br>
<br>
<br>
[root@ovirt2 test ~]# mount -t glusterfs
ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/<br>
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s<br>
<br>
That's more how I expected, it is pointing to glusterfs
performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:27 PM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:56BD18D7.8080909@j2.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
don't know if it helps, but I ran a few more tests, all from the
same hardware node.<br>
<br>
The VM:<br>
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s<br>
<br>
Writing directly to gluster volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000
oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s<br>
<br>
<br>
Writing to NFS volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s<br>
<br>
NFS & Gluster are using the same interface. Tests were not
run at same time.<br>
<br>
This would suggest my problem isn't glusterfs, but the VM
performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:13 PM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:56BD1589.8090506@j2.com"
type="cite">xml
attached. <br>
<br>
<br>
On 02/11/2016 12:28 PM, Nir Soffer wrote: <br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 8:27 PM, Bill
James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote: <br>
<blockquote type="cite">thank you for the reply. <br>
<br>
We setup gluster using the names associated with NIC 2
IP. <br>
Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
<br>
That's NIC 2's IP. <br>
Using 'iftop -i eno2 -L 5 -t' : <br>
<br>
dd if=/dev/zero of=/root/testfile bs=1M count=1000
oflag=direct <br>
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s <br>
</blockquote>
Can you share the xml of this vm? You can find it in vdsm
log, <br>
at the time you start the vm. <br>
<br>
Or you can do (on the host): <br>
<br>
# virsh <br>
virsh # list <br>
(username: vdsm@ovirt password: shibboleth) <br>
virsh # dumpxml vm-id <br>
<br>
<blockquote type="cite">Peak rate
(sent/received/total): 281Mb 5.36Mb <br>
282Mb <br>
Cumulative (sent/received/total):
1.96GB 14.6MB <br>
1.97GB <br>
<br>
gluster volume info gv1: <br>
Options Reconfigured: <br>
performance.write-behind-window-size: 4MB <br>
performance.readdir-ahead: on <br>
performance.cache-size: 1GB <br>
performance.write-behind: off <br>
<br>
performance.write-behind: off didn't help. <br>
Neither did any other changes I've tried. <br>
<br>
<br>
There is no VM traffic on this VM right now except my
test. <br>
<br>
<br>
<br>
On 02/10/2016 11:55 PM, Nir Soffer wrote: <br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 2:42 AM,
Ravishankar N <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a>
<br>
wrote: <br>
<blockquote type="cite">+gluster-users <br>
<br>
Does disabling 'performance.write-behind' give a
better throughput? <br>
<br>
<br>
<br>
On 02/10/2016 11:06 PM, Bill James wrote: <br>
<blockquote type="cite">I'm setting up a ovirt
cluster
using glusterfs and noticing not stellar <br>
performance. <br>
Maybe my setup could use some adjustments? <br>
<br>
3 hardware nodes running centos7.2, glusterfs
3.7.6.1, ovirt 3.6.2.6-1. <br>
Each node has 8 spindles configured in 1 array which
is split using LVM <br>
with one logical volume for system and one for
gluster. <br>
They each have 4 NICs, <br>
NIC1 = ovirtmgmt <br>
NIC2 = gluster (1GbE) <br>
</blockquote>
</blockquote>
How do you ensure that gluster trafic is using this nic?
<br>
<br>
<blockquote type="cite">
<blockquote type="cite"> NIC3 = VM traffic <br>
</blockquote>
</blockquote>
How do you ensure that vm trafic is using this nic? <br>
<br>
<blockquote type="cite">
<blockquote type="cite">I tried with default glusterfs
settings <br>
</blockquote>
</blockquote>
And did you find any difference? <br>
<br>
<blockquote type="cite">
<blockquote type="cite">and also with: <br>
performance.cache-size: 1GB <br>
performance.readdir-ahead: on <br>
performance.write-behind-window-size: 4MB <br>
<br>
[root@ovirt3 test scripts]# gluster volume info gv1
<br>
<br>
Volume Name: gv1 <br>
Type: Replicate <br>
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 <br>
Status: Started <br>
Number of Bricks: 1 x 3 = 3 <br>
Transport-type: tcp <br>
Bricks: <br>
Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Options Reconfigured: <br>
performance.cache-size: 1GB <br>
performance.readdir-ahead: on <br>
performance.write-behind-window-size: 4MB <br>
<br>
<br>
Using simple dd test on VM in ovirt: <br>
dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct <br>
</blockquote>
</blockquote>
block size of 1G?! <br>
<br>
Try 1M (our default for storage operations) <br>
<br>
<blockquote type="cite">
<blockquote type="cite"> 1073741824 bytes (1.1
GB) copied, 65.9337 s, 16.3 MB/s <br>
<br>
Another VM not in ovirt using nfs: <br>
dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct <br>
1073741824 bytes (1.1 GB) copied, 27.0079 s,
39.8 MB/s <br>
<br>
<br>
Is that expected or is there a better way to set it
up to get better <br>
performance? <br>
</blockquote>
</blockquote>
Adding Niels for advice. <br>
<br>
<blockquote type="cite">
<blockquote type="cite">This email, its contents and
.... <br>
</blockquote>
</blockquote>
Please avoid this, this is a public mailing list,
everything you write <br>
here is public. <br>
<br>
Nir <br>
</blockquote>
I'll have to look into how to remove this sig for this
mailing list.... <br>
<br>
Cloud Services for Business <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="http://www.j2.com">www.j2.com</a> <br>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe |
Onebox <br>
<br>
<br>
This email, its contents and attachments contain
information from j2 Global, <br>
Inc. and/or its affiliates which may be privileged,
confidential or <br>
otherwise protected from disclosure. The information is
intended to be for <br>
the addressee(s) only. If you are not an addressee, any
disclosure, copy, <br>
distribution, or use of the contents of this message is
prohibited. If you <br>
have received this email in error please notify the sender
by reply e-mail <br>
and delete the original message and any copies. (c) 2015
j2 Global, Inc. All <br>
rights reserved. eFax, eVoice, Campaigner, FuseMail,
KeepItSafe, and Onebox <br>
are registered trademarks of j2 Global, Inc. and its
affiliates. <br>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
style="color:windowtext; text-decoration:none"><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" height="46" width="391"
border="0"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain information from
<a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of
this message is prohibited. If you have received this email
in error please notify the sender by reply e-mail and delete
the original message and any copies. © 2015 <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/">eFax ®</a>,
<a moz-do-not-send="true"
href="http://www.evoice.com/">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner ®</a>,
<a
moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and
<a
moz-do-not-send="true"
href="http://www.onebox.com/">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global, Inc</a>. and its affiliates.</span></p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
style="color:windowtext;
text-decoration:none"><img moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" height="46" width="391"
border="0"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of this
message is prohibited. If you have received this email in
error please notify the sender by reply e-mail and delete the
original message and any copies. © 2015 <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global,
Inc</a>. All rights reserved. <a moz-do-not-send="true"
href="http://www.efax.com/">eFax ®</a>, <a
moz-do-not-send="true"
href="http://www.evoice.com/">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner ®</a>, <a
moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and
<a
moz-do-not-send="true"
href="http://www.onebox.com/">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global,
Inc</a>. and its affiliates.</span></p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<br>
</body>
</html>
--------------020701000405040103030307--
6:41 p.m.
--------------080509040202050109010204
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 8bit
wow, that made a whole lot of difference!
Thank you!
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile1 bs=1M
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 20.2778 s, 51.7 MB/s
these are the options now for the record.
Options Reconfigured:
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
performance.write-behind: off
performance.write-behind-window-size: 4MB
performance.cache-size: 1GB
performance.readdir-ahead: on
Thanks again!
On 2/11/16 8:18 PM, Ravishankar N wrote:
Hi Bill,
Can you enable virt-profile setting for your volume and see if that
helps? You need to enable this optimization when you create the volume
using ovrit, or use the following command for an existing volume:
#gluster volume set <volname> group virt
-Ravi
On 02/12/2016 05:22 AM, Bill James wrote:
> My apologies, I'm showing how much of a noob I am.
> Ignore last direct to gluster numbers, as that wasn't really glusterfs.
>
>
> [root@ovirt2 test ~]# mount -t glusterfs
> ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/
> [root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
> bs=1M count=1000 oflag=direct
> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
>
> That's more how I expected, it is pointing to glusterfs performance.
>
>
>
> On 02/11/2016 03:27 PM, Bill James wrote:
>> don't know if it helps, but I ran a few more tests, all from the
>> same hardware node.
>>
>> The VM:
>> [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
>> count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
>>
>> Writing directly to gluster volume:
>> [root@ovirt2 test ~]# time dd if=/dev/zero
>> of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
>>
>>
>> Writing to NFS volume:
>> [root@ovirt2 test ~]# time dd if=/dev/zero
>> of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
>>
>> NFS & Gluster are using the same interface. Tests were not run at
>> same time.
>>
>> This would suggest my problem isn't glusterfs, but the VM performance.
>>
>>
>>
>> On 02/11/2016 03:13 PM, Bill James wrote:
>>> xml attached.
>>>
>>>
>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:
>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com>
>>>> wrote:
>>>>> thank you for the reply.
>>>>>
>>>>> We setup gluster using the names associated with NIC 2 IP.
>>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>
>>>>> That's NIC 2's IP.
>>>>> Using 'iftop -i eno2 -L 5 -t' :
>>>>>
>>>>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>>>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
>>>> Can you share the xml of this vm? You can find it in vdsm log,
>>>> at the time you start the vm.
>>>>
>>>> Or you can do (on the host):
>>>>
>>>> # virsh
>>>> virsh # list
>>>> (username: vdsm@ovirt password: shibboleth)
>>>> virsh # dumpxml vm-id
>>>>
>>>>> Peak rate (sent/received/total): 281Mb 5.36Mb
>>>>> 282Mb
>>>>> Cumulative (sent/received/total): 1.96GB 14.6MB
>>>>> 1.97GB
>>>>>
>>>>> gluster volume info gv1:
>>>>> Options Reconfigured:
>>>>> performance.write-behind-window-size: 4MB
>>>>> performance.readdir-ahead: on
>>>>> performance.cache-size: 1GB
>>>>> performance.write-behind: off
>>>>>
>>>>> performance.write-behind: off didn't help.
>>>>> Neither did any other changes I've tried.
>>>>>
>>>>>
>>>>> There is no VM traffic on this VM right now except my test.
>>>>>
>>>>>
>>>>>
>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
>>>>>> <ravishankar(a)redhat.com>
>>>>>> wrote:
>>>>>>> +gluster-users
>>>>>>>
>>>>>>> Does disabling 'performance.write-behind' give a
better
>>>>>>> throughput?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>>>>> I'm setting up a ovirt cluster using glusterfs and
noticing
>>>>>>>> not stellar
>>>>>>>> performance.
>>>>>>>> Maybe my setup could use some adjustments?
>>>>>>>>
>>>>>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1,
ovirt
>>>>>>>> 3.6.2.6-1.
>>>>>>>> Each node has 8 spindles configured in 1 array which is
split
>>>>>>>> using LVM
>>>>>>>> with one logical volume for system and one for gluster.
>>>>>>>> They each have 4 NICs,
>>>>>>>> NIC1 = ovirtmgmt
>>>>>>>> NIC2 = gluster (1GbE)
>>>>>> How do you ensure that gluster trafic is using this nic?
>>>>>>
>>>>>>>> NIC3 = VM traffic
>>>>>> How do you ensure that vm trafic is using this nic?
>>>>>>
>>>>>>>> I tried with default glusterfs settings
>>>>>> And did you find any difference?
>>>>>>
>>>>>>>> and also with:
>>>>>>>> performance.cache-size: 1GB
>>>>>>>> performance.readdir-ahead: on
>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>
>>>>>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>>>>>
>>>>>>>> Volume Name: gv1
>>>>>>>> Type: Replicate
>>>>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>>>>> Status: Started
>>>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>>>> Transport-type: tcp
>>>>>>>> Bricks:
>>>>>>>> Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>>> Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>>> Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>>> Options Reconfigured:
>>>>>>>> performance.cache-size: 1GB
>>>>>>>> performance.readdir-ahead: on
>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>
>>>>>>>>
>>>>>>>> Using simple dd test on VM in ovirt:
>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>> block size of 1G?!
>>>>>>
>>>>>> Try 1M (our default for storage operations)
>>>>>>
>>>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3
MB/s
>>>>>>>>
>>>>>>>> Another VM not in ovirt using nfs:
>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8
MB/s
>>>>>>>>
>>>>>>>>
>>>>>>>> Is that expected or is there a better way to set it up to
get
>>>>>>>> better
>>>>>>>> performance?
>>>>>> Adding Niels for advice.
>>>>>>
>>>>>>>> This email, its contents and ....
>>>>>> Please avoid this, this is a public mailing list, everything you
>>>>>> write
>>>>>> here is public.
>>>>>>
>>>>>> Nir
>>>>> I'll have to look into how to remove this sig for this mailing
>>>>> list....
>>>>>
>>>>> Cloud Services for Business www.j2.com
>>>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>>>>
>>>>>
>>>>> This email, its contents and attachments contain information from
>>>>> j2 Global,
>>>>> Inc. and/or its affiliates which may be privileged, confidential or
>>>>> otherwise protected from disclosure. The information is intended
>>>>> to be for
>>>>> the addressee(s) only. If you are not an addressee, any
>>>>> disclosure, copy,
>>>>> distribution, or use of the contents of this message is
>>>>> prohibited. If you
>>>>> have received this email in error please notify the sender by
>>>>> reply e-mail
>>>>> and delete the original message and any copies. (c) 2015 j2
>>>>> Global, Inc. All
>>>>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe,
>>>>> and Onebox
>>>>> are registered trademarks of j2 Global, Inc. and its affiliates.
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>> www.j2.com
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>
>> This email, its contents and attachments contain information from j2
>> Global, Inc
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
>> and/or its affiliates which may be privileged, confidential or
>> otherwise protected from disclosure. The information is intended to
>> be for the addressee(s) only. If you are not an addressee, any
>> disclosure, copy, distribution, or use of the contents of this
>> message is prohibited. If you have received this email in error
>> please notify the sender by reply e-mail and delete the original
>> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;.
>> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ®
>> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
>> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
>> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/>
>> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
>> and its affiliates.
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> www.j2.com
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>
> This email, its contents and attachments contain information from j2
> Global, Inc
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
> and/or its affiliates which may be privileged, confidential or
> otherwise protected from disclosure. The information is intended to
> be for the addressee(s) only. If you are not an addressee, any
> disclosure, copy, distribution, or use of the contents of this
> message is prohibited. If you have received this email in error
> please notify the sender by reply e-mail and delete the original
> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;.
> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ®
> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/>
> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
> and its affiliates.
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.
--------------080509040202050109010204
Content-Type: text/html; charset="windows-1252"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
wow, that made a whole lot of difference!<br>
Thank you!<br>
<br>
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile1 bs=1M
count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 20.2778 s, 51.7 MB/s<br>
<br>
these are the options now for the record.<br>
<br>
Options Reconfigured:<br>
cluster.server-quorum-type: server<br>
cluster.quorum-type: auto<br>
network.remote-dio: enable<br>
cluster.eager-lock: enable<br>
performance.stat-prefetch: off<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
performance.write-behind: off<br>
performance.write-behind-window-size: 4MB<br>
performance.cache-size: 1GB<br>
performance.readdir-ahead: on<br>
<br>
<br>
Thanks again!<br>
<br>
<br>
<div class="moz-cite-prefix">On 2/11/16 8:18 PM, Ravishankar N
wrote:<br>
</div>
<blockquote cite="mid:56BD5D15.6060607@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div class="moz-cite-prefix">Hi Bill,<br>
Can you enable virt-profile setting for your volume and see if
that helps? You need to enable this optimization when you create
the volume using ovrit, or use the following command for an
existing volume:<br>
<tt><br>
#gluster volume set <volname> group virt</tt><br>
<br>
-Ravi<br>
<br>
<br>
On 02/12/2016 05:22 AM, Bill James wrote:<br>
</div>
<blockquote cite="mid:56BD1EB9.5000802@j2.com"
type="cite"> My
apologies, I'm showing how much of a noob I am.<br>
Ignore last direct to gluster numbers, as that wasn't really
glusterfs.<br>
<br>
<br>
[root@ovirt2 test ~]# mount -t glusterfs
ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/<br>
[root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s<br>
<br>
That's more how I expected, it is pointing to glusterfs
performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:27 PM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:56BD18D7.8080909@j2.com"
type="cite">
don't know if it helps, but I ran a few more tests, all from
the same hardware node.<br>
<br>
The VM:<br>
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile
bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s<br>
<br>
Writing directly to gluster volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M count=1000
oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s<br>
<br>
<br>
Writing to NFS volume:<br>
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct<br>
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s<br>
<br>
NFS & Gluster are using the same interface. Tests were not
run at same time.<br>
<br>
This would suggest my problem isn't glusterfs, but the VM
performance.<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 02/11/2016 03:13 PM, Bill
James wrote:<br>
</div>
<blockquote cite="mid:56BD1589.8090506@j2.com"
type="cite">xml
attached. <br>
<br>
<br>
On 02/11/2016 12:28 PM, Nir Soffer wrote: <br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 8:27 PM,
Bill James <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote: <br>
<blockquote type="cite">thank you for the reply. <br>
<br>
We setup gluster using the names associated with NIC 2
IP. <br>
Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
<br>
That's NIC 2's IP. <br>
Using 'iftop -i eno2 -L 5 -t' : <br>
<br>
dd if=/dev/zero of=/root/testfile bs=1M count=1000
oflag=direct <br>
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s <br>
</blockquote>
Can you share the xml of this vm? You can find it in vdsm
log, <br>
at the time you start the vm. <br>
<br>
Or you can do (on the host): <br>
<br>
# virsh <br>
virsh # list <br>
(username: vdsm@ovirt password: shibboleth) <br>
virsh # dumpxml vm-id <br>
<br>
<blockquote type="cite">Peak rate
(sent/received/total): 281Mb 5.36Mb
<br>
282Mb <br>
Cumulative (sent/received/total):
1.96GB 14.6MB <br>
1.97GB <br>
<br>
gluster volume info gv1: <br>
Options Reconfigured: <br>
performance.write-behind-window-size: 4MB <br>
performance.readdir-ahead: on <br>
performance.cache-size: 1GB <br>
performance.write-behind: off <br>
<br>
performance.write-behind: off didn't help. <br>
Neither did any other changes I've tried. <br>
<br>
<br>
There is no VM traffic on this VM right now except my
test. <br>
<br>
<br>
<br>
On 02/10/2016 11:55 PM, Nir Soffer wrote: <br>
<blockquote type="cite">On Thu, Feb 11, 2016 at 2:42 AM,
Ravishankar N <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a>
<br>
wrote: <br>
<blockquote type="cite">+gluster-users <br>
<br>
Does disabling 'performance.write-behind' give a
better throughput? <br>
<br>
<br>
<br>
On 02/10/2016 11:06 PM, Bill James wrote: <br>
<blockquote type="cite">I'm setting up a ovirt
cluster using glusterfs and noticing not stellar <br>
performance. <br>
Maybe my setup could use some adjustments? <br>
<br>
3 hardware nodes running centos7.2, glusterfs
3.7.6.1, ovirt 3.6.2.6-1. <br>
Each node has 8 spindles configured in 1 array
which is split using LVM <br>
with one logical volume for system and one for
gluster. <br>
They each have 4 NICs, <br>
NIC1 = ovirtmgmt <br>
NIC2 = gluster (1GbE) <br>
</blockquote>
</blockquote>
How do you ensure that gluster trafic is using this
nic? <br>
<br>
<blockquote type="cite">
<blockquote type="cite"> NIC3 = VM traffic
<br>
</blockquote>
</blockquote>
How do you ensure that vm trafic is using this nic? <br>
<br>
<blockquote type="cite">
<blockquote type="cite">I tried with default
glusterfs settings <br>
</blockquote>
</blockquote>
And did you find any difference? <br>
<br>
<blockquote type="cite">
<blockquote type="cite">and also with: <br>
performance.cache-size: 1GB <br>
performance.readdir-ahead: on <br>
performance.write-behind-window-size: 4MB <br>
<br>
[root@ovirt3 test scripts]# gluster volume info
gv1 <br>
<br>
Volume Name: gv1 <br>
Type: Replicate <br>
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 <br>
Status: Started <br>
Number of Bricks: 1 x 3 = 3 <br>
Transport-type: tcp <br>
Bricks: <br>
Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
<br>
Options Reconfigured: <br>
performance.cache-size: 1GB <br>
performance.readdir-ahead: on <br>
performance.write-behind-window-size: 4MB <br>
<br>
<br>
Using simple dd test on VM in ovirt: <br>
dd if=/dev/zero of=/root/testfile bs=1G
count=1 oflag=direct <br>
</blockquote>
</blockquote>
block size of 1G?! <br>
<br>
Try 1M (our default for storage operations) <br>
<br>
<blockquote type="cite">
<blockquote type="cite"> 1073741824 bytes
(1.1 GB) copied, 65.9337 s, 16.3 MB/s <br>
<br>
Another VM not in ovirt using nfs: <br>
dd if=/dev/zero of=/root/testfile bs=1G
count=1 oflag=direct <br>
1073741824 bytes (1.1 GB) copied, 27.0079
s, 39.8 MB/s <br>
<br>
<br>
Is that expected or is there a better way to set
it up to get better <br>
performance? <br>
</blockquote>
</blockquote>
Adding Niels for advice. <br>
<br>
<blockquote type="cite">
<blockquote type="cite">This email, its contents and
.... <br>
</blockquote>
</blockquote>
Please avoid this, this is a public mailing list,
everything you write <br>
here is public. <br>
<br>
Nir <br>
</blockquote>
I'll have to look into how to remove this sig for this
mailing list.... <br>
<br>
Cloud Services for Business <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="http://www.j2.com">www.j2.com</a> <br>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe
| Onebox <br>
<br>
<br>
This email, its contents and attachments contain
information from j2 Global, <br>
Inc. and/or its affiliates which may be privileged,
confidential or <br>
otherwise protected from disclosure. The information is
intended to be for <br>
the addressee(s) only. If you are not an addressee, any
disclosure, copy, <br>
distribution, or use of the contents of this message is
prohibited. If you <br>
have received this email in error please notify the
sender by reply e-mail <br>
and delete the original message and any copies. (c) 2015
j2 Global, Inc. All <br>
rights reserved. eFax, eVoice, Campaigner, FuseMail,
KeepItSafe, and Onebox <br>
are registered trademarks of j2 Global, Inc. and its
affiliates. <br>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
style="color:windowtext; text-decoration:none"><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" height="46" border="0"
width="391"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain information
from <a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of
this message is prohibited. If you have received this
email in error please notify the sender by reply e-mail
and delete the original message and any copies. © 2015 <a
moz-do-not-send="true"
href="http://www.j2.com/">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/">eFax
®</a>, <a moz-do-not-send="true"
href="http://www.evoice.com/">eVoice ®</a>, <a
moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner
®</a>, <a moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail ®</a>, <a
moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe
®</a> and <a moz-do-not-send="true"
href="http://www.onebox.com/">Onebox ®</a> are r
egistered trademarks of <a moz-do-not-send="true"
href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
style="color:windowtext; text-decoration:none"><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" height="46" border="0"
width="391"></span></a></p>
<p><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray">This
email, its contents and attachments contain information from
<a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of
this message is prohibited. If you have received this email
in error please notify the sender by reply e-mail and delete
the original message and any copies. © 2015 <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/">eFax ®</a>,
<a moz-do-not-send="true"
href="http://www.evoice.com/">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner ®</a>,
<a
moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and
<a
moz-do-not-send="true"
href="http://www.onebox.com/">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true" href="http://www.j2.com/">j2
Global, Inc</a>. and its affiliates.</span></p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<br>
</blockquote>
<br>
<p><a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
style='color:windowtext;
text-decoration:none'><img border=0 width=391 height=46
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com"></span></a></p>
<p><span
style='font-size:8.0pt;font-family:"Arial","sans-serif";
color:gray'>This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</span></p></body>
</html>
--------------080509040202050109010204--
6:56 p.m.
On 02/12/2016 09:11 PM, Bill James wrote:
wow, that made a whole lot of difference!
Thank you!
[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile1 bs=1M
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 20.2778 s, 51.7 MB/s
That's great. It was
Vijay Bellur who noticed that it was not enabled on
your volume while we were talking on irc. So thanks to him.
11:38 a.m.
--Apple-Mail=_F4D45E80-CC6B-492F-BC3F-6304354C342D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
Hi Ravi and colleagues
(apologies for hijacking this thread but I=92m not sure where else to repor=
t this (and it is related).)
With gluster 3.7.10, running
#gluster volume set <volname> group virt
fails with:
volume set: failed: option : eager-lock does not exist
Did you mean eager-lock?
I had to remove the eager-lock setting from /var/lib/glusterd/groups/virt t=
o get this to work. It seems like setting eager-lock has been removed from =
latest gluster. Is this correct? Either way, is there anything else I shoul=
d do?
Cheers,
Roderick
On 12 Feb 2016, at 6:18 AM, Ravishankar N
<ravishankar(a)redhat.com> wrote:
=20
Hi Bill,
Can you enable virt-profile setting for your volume and see if that helps=
? You
need to enable this optimization when you create the volume using ovr=
it, or use the following command for an existing volume:
=20
#gluster volume set <volname> group virt
=20
-Ravi
=20
=20
On 02/12/2016 05:22 AM, Bill James wrote:
> My apologies, I'm showing how much of a noob I am.
> Ignore last direct to gluster numbers, as that wasn't really glusterfs.
>=20
>=20
> [root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com:/gv1 /=
mnt/tmp/
> [root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/mnt/tmp/testfile2 bs=
=3D1M count=3D1000 oflag=3Ddirect
> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
>=20
> That's more how I expected, it is pointing to glusterfs performance.
>=20
>=20
>=20
> On 02/11/2016 03:27 PM, Bill James wrote:
>> don't know if it helps, but I ran a few more tests, all from the same h=
ardware node.
>>=20
>> The VM:
>> [root@billjov1 ~]# time dd if=3D/dev/zero of=3D/root/testfile bs=3D1M c=
ount=3D1000 oflag=3Ddirect
>> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
>>=20
>> Writing directly to gluster volume:
>> [root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/gluster-store/brick1=
/gv1/testfile bs=3D1M count=3D1000 oflag=3Ddirect
>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
>>=20
>>=20
>> Writing to NFS volume:
>> [root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/mnt/storage/qa/testf=
ile bs=3D1M count=3D1000 oflag=3Ddirect
>> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
>>=20
>> NFS & Gluster are using the same interface. Tests were not run at same =
time.
>>=20
>> This would suggest my problem isn't glusterfs, but the VM performance.
>>=20
>>=20
>>=20
>> On 02/11/2016 03:13 PM, Bill James wrote:
>>> xml attached.=20
>>>=20
>>>=20
>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:=20
>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com>
<mail=
to:bill.james@j2.com> wrote:=20
>>>>> thank you for the reply.=20
>>>>>=20
>>>>> We setup gluster using the names associated with NIC 2 IP.=20
>>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1=20
>>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1=20
>>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1=20
>>>>>=20
>>>>> That's NIC 2's IP.=20
>>>>> Using 'iftop -i eno2 -L 5 -t' :=20
>>>>>=20
>>>>> dd if=3D/dev/zero of=3D/root/testfile bs=3D1M count=3D1000 oflag=3Dd=
irect=20
>>>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4
MB/s=20
>>>> Can you share the xml of this vm? You can find it in vdsm log,=20
>>>> at the time you start the vm.=20
>>>>=20
>>>> Or you can do (on the host):=20
>>>>=20
>>>> # virsh=20
>>>> virsh # list=20
>>>> (username: vdsm@ovirt password: shibboleth)=20
>>>> virsh # dumpxml vm-id=20
>>>>=20
>>>>> Peak rate (sent/received/total): 281Mb 5.36Mb=
=20
>>>>> 282Mb=20
>>>>> Cumulative (sent/received/total): 1.96GB 14.6MB=
=20
>>>>> 1.97GB=20
>>>>>=20
>>>>> gluster volume info gv1:=20
>>>>> Options Reconfigured:=20
>>>>> performance.write-behind-window-size: 4MB=20
>>>>> performance.readdir-ahead: on=20
>>>>> performance.cache-size: 1GB=20
>>>>> performance.write-behind: off=20
>>>>>=20
>>>>> performance.write-behind: off didn't help.=20
>>>>> Neither did any other changes I've tried.=20
>>>>>=20
>>>>>=20
>>>>> There is no VM traffic on this VM right now except my test.=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:=20
>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
<ravishankar(a)redhat.=
com> <mailto:ravishankar@redhat.com>=20
>>>>>> wrote:=20
>>>>>>> +gluster-users=20
>>>>>>>=20
>>>>>>> Does disabling 'performance.write-behind' give a
better throughput=
?=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:=20
>>>>>>>> I'm setting up a ovirt cluster using glusterfs and
noticing not s=
tellar=20
>>>>>>>> performance.=20
>>>>>>>> Maybe my setup could use some adjustments?=20
>>>>>>>>=20
>>>>>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1,
ovirt 3.6.=
2.6-1.=20
>>>>>>>> Each node has 8 spindles configured
in 1 array which is split usi=
ng LVM=20
>>>>>>>> with one logical volume for system
and one for gluster.=20
>>>>>>>> They each have 4 NICs,=20
>>>>>>>> NIC1 =3D ovirtmgmt=20
>>>>>>>> NIC2 =3D gluster (1GbE)=20
>>>>>> How do you ensure that gluster trafic is using this nic?=20
>>>>>>=20
>>>>>>>> NIC3 =3D VM traffic=20
>>>>>> How do you ensure that vm trafic is using this nic?=20
>>>>>>=20
>>>>>>>> I tried with default glusterfs settings=20
>>>>>> And did you find any difference?=20
>>>>>>=20
>>>>>>>> and also with:=20
>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>> performance.write-behind-window-size: 4MB=20
>>>>>>>>=20
>>>>>>>> [root@ovirt3 test scripts]# gluster volume info gv1=20
>>>>>>>>=20
>>>>>>>> Volume Name: gv1=20
>>>>>>>> Type: Replicate=20
>>>>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2=20
>>>>>>>> Status: Started=20
>>>>>>>> Number of Bricks: 1 x 3 =3D 3=20
>>>>>>>> Transport-type: tcp=20
>>>>>>>> Bricks:=20
>>>>>>>> Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1=20
>>>>>>>> Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1=20
>>>>>>>> Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1=20
>>>>>>>> Options Reconfigured:=20
>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>> performance.write-behind-window-size: 4MB=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> Using simple dd test on VM in ovirt:=20
>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile bs=3D1G
count=3D1 oflag=
=3Ddirect=20
>>>>>> block size of 1G?!=20
>>>>>>=20
>>>>>> Try 1M (our default for storage operations)=20
>>>>>>=20
>>>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3
MB/s=20
>>>>>>>>=20
>>>>>>>> Another VM not in ovirt using nfs:=20
>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile bs=3D1G
count=3D1 ofla=
g=3Ddirect=20
>>>>>>>> 1073741824 bytes (1.1 GB)
copied, 27.0079 s, 39.8 MB/s=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> Is that expected or is there a better way to set it up to
get bet=
ter=20
>>>>>>>> performance?=20
>>>>>> Adding Niels for advice.=20
>>>>>>=20
>>>>>>>> This email, its contents and ....=20
>>>>>> Please avoid this, this is a public mailing list, everything you
wr=
ite=20
>>>>>> here is public.=20
>>>>>>=20
>>>>>> Nir=20
>>>>> I'll have to look into how to remove this sig for this mailing
list.=
...=20
>>>>>=20
>>>>> Cloud Services for Business www.j2.com <http://www.j2.com/>=20
>>>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox=20
>>>>>=20
>>>>>=20
>>>>> This email, its contents and attachments contain information from j2=
Global,=20
>>>>> Inc. and/or its affiliates which may be
privileged, confidential or=
=20
>>>>> otherwise protected from disclosure. The
information is intended to =
be for=20
>>>>> the addressee(s) only. If you are not an
addressee, any disclosure, =
copy,=20
>>>>> distribution, or use of the contents of this
message is prohibited. =
If you=20
>>>>> have received this email in error please notify
the sender by reply =
e-mail=20
>>>>> and delete the original message and any copies.
(c) 2015 j2 Global, =
Inc. All=20
>>>>> rights reserved. eFax, eVoice, Campaigner,
FuseMail, KeepItSafe, and=
Onebox=20
>>>>> are registered trademarks of j2 Global, Inc. and
its affiliates.=20
>>>=20
>>>=20
>>>=20
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/=
mailman/listinfo/users>
utm_campaign=3Demployeeemail>
>> This email, its contents and attachments contain information
from j2 Gl=
obal, Inc
<http://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-refe=
rral&utm_campaign=3Demployemail>. and/or its affiliates which may be privil=
eged, confidential or otherwise protected from disclosure. The information =
is intended to be for the addressee(s) only. If you are not an addressee, a=
ny disclosure, copy, distribution, or use of the contents of this message i=
s prohibited. If you have received this email in error please notify the se=
nder by reply e-mail and delete the original message and any copies. =A9 20=
15 j2 Global, Inc <http://www.j2.com/>;. All rights reserved. eFax =AE <http=
://www.efax.com/>, eVoice =AE <http://www.evoice.com/>;, Campaigner =AE <htt=
p://www.campaigner.com/>;, FuseMail =AE <http://www.fusemail.com/>;, KeepItSa=
fe =AE <http://www.keepitsafe.com/> and Onebox =AE <http://www.onebox.com/>=
are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. and its=
affiliates.
>>=20
>>=20
>>=20
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/m=
ailman/listinfo/users>
tm_campaign=3Demployeeemail>
> This email, its contents and attachments contain information from
j2 Glo=
bal, Inc
<http://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-refer=
ral&utm_campaign=3Demployemail>. and/or its affiliates which may be privile=
ged, confidential or otherwise protected from disclosure. The information i=
s intended to be for the addressee(s) only. If you are not an addressee, an=
y disclosure, copy, distribution, or use of the contents of this message is=
prohibited. If you have received this email in error please notify the sen=
der by reply e-mail and delete the original message and any copies. =A9 201=
5 j2 Global, Inc <http://www.j2.com/>;. All rights reserved. eFax =AE <http:=
//www.efax.com/>, eVoice =AE <http://www.evoice.com/>;, Campaigner =AE <http=
://www.campaigner.com/>, FuseMail =AE <http://www.fusemail.com/>;, KeepItSaf=
e =AE <http://www.keepitsafe.com/> and Onebox =AE <http://www.onebox.com/> =
are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. and its =
affiliates.
>=20
>=20
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/ma=
ilman/listinfo/users>
=20
=20
=20
--=20
This message is subject to the CSIR's copyright terms and conditions, e-m=
ail
legal notice, and implemented Open Document Format (ODF) standard.=20
The full disclaimer details can be found at
http://www.csir.co.za/disclai=
mer.html
<http://www.csir.co.za/disclaimer.html>;.
=20
This message has been scanned for viruses and dangerous content by MailSc=
anner
<http://www.mailscanner.info/>,=20
and is believed to be clean.
=20
=20
Please consider the environment before printing this email.
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--=20
This message is subject to the CSIR's copyright terms and conditions, e-mai=
l legal notice, and implemented Open Document Format (ODF) standard.=20
The full disclaimer details can be found at http://www.csir.co.za/disclaime=
r.html.
This message has been scanned for viruses and dangerous content by MailScan=
ner,=20
and is believed to be clean.
Please consider the environment before printing this email.
--Apple-Mail=_F4D45E80-CC6B-492F-BC3F-6304354C342D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html charset=
=3Dwindows-1252"></head><body style=3D"word-wrap: break-word;
-webkit-nbsp-=
mode: space; -webkit-line-break: after-white-space;" class=3D"">Hi Ravi
and=
colleagues<div class=3D""><br
class=3D""></div><div class=3D"">(apologies =
for hijacking this thread but I=92m not sure where else to report this (and=
it is related).)<div class=3D""><br
class=3D""></div><div class=3D"">With =
gluster 3.7.10, running</div><div class=3D"">#gluster volume set
<volnam=
e> group virt</div><div class=3D"">fails
with:</div><div class=3D"">volu=
me set: failed: option : eager-lock does not exist</div><div
class=3D"">Did=
you mean eager-lock?</div><div class=3D""><br
class=3D""></div><div class=
=3D"">I had to remove the eager-lock setting from /var/lib/glusterd/groups/=
virt to get this to work. It seems like setting eager-lock has been removed=
from latest gluster. Is this correct? Either way, is there anything else I=
should do?</div><div class=3D""><br
class=3D""></div><div class=3D"">Cheer=
s,</div><div class=3D""><br
class=3D""></div><div class=3D"">Roderick<br cl=
ass=3D""><div class=3D""><br
class=3D""><div><blockquote type=3D"cite" clas=
s=3D""><div class=3D"">On 12 Feb 2016, at 6:18 AM,
Ravishankar N <<a hre=
f=3D"mailto:ravishankar@redhat.com"
class=3D"">ravishankar(a)redhat.com</a>&g=
t; wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D"">
=20=20
<meta content=3D"text/html; charset=3Dwindows-1252"
http-equiv=3D"Conte=
nt-Type" class=3D"">
=20=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">
<div class=3D"moz-cite-prefix">Hi Bill,<br
class=3D"">
Can you enable virt-profile setting for your volume and see if
that helps? You need to enable this optimization when you create
the volume using ovrit, or use the following command for an
existing volume:<br class=3D"">
<tt class=3D""><br class=3D"">
#gluster volume set <volname> group virt</tt><br
class=3D"">
<br class=3D"">
-Ravi<br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/12/2016 05:22 AM, Bill James wrote:<br class=3D"">
</div>
<blockquote cite=3D"mid:56BD1EB9.5000802@j2.com" type=3D"cite"
class=3D=
"">
<meta content=3D"text/html; charset=3Dwindows-1252"
http-equiv=3D"Con=
tent-Type" class=3D"">
My apologies, I'm showing how much of a noob I am.<br
class=3D"">
Ignore last direct to gluster numbers, as that wasn't really
glusterfs.<br class=3D"">
<br class=3D"">
<br class=3D"">
[root@ovirt2 test ~]# mount -t glusterfs
<a href=3D"http://ovirt2-ks.test.j2noc.com"
class=3D"">ovirt2-ks.test=
.j2noc.com</a>:/gv1 /mnt/tmp/<br class=3D"">
[root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/mnt/tmp/testfile2
bs=3D1M count=3D1000 oflag=3Ddirect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s<br
class=3D"">
<br class=3D"">
That's more how I expected, it is pointing to glusterfs
performance.<br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 02/11/2016 03:27 PM, Bill James
wrote:<br class=3D"">
</div>
<blockquote cite=3D"mid:56BD18D7.8080909@j2.com"
type=3D"cite" class=
=3D"">
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3Dwindows-1252" class=3D"">
don't know if it helps, but I ran a few more tests, all from the
same hardware node.<br class=3D"">
<br class=3D"">
The VM:<br class=3D"">
[root@billjov1 ~]# time dd if=3D/dev/zero of=3D/root/testfile bs=3D=
1M
count=3D1000 oflag=3Ddirect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s<br class=3D"=
">
<br class=3D"">
Writing directly to gluster volume:<br class=3D"">
[root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/gluster-store/brick1/gv1/testfile bs=3D1M count=3D1000
oflag=3Ddirect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s<br
class=3D"">
<br class=3D"">
<br class=3D"">
Writing to NFS volume:<br class=3D"">
[root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/mnt/storage/qa/testfile bs=3D1M count=3D1000 oflag=3Ddirect<b=
r class=3D"">
1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s<br class=3D"=
">
<br class=3D"">
NFS & Gluster are using the same interface. Tests were not
run at same time.<br class=3D"">
<br class=3D"">
This would suggest my problem isn't glusterfs, but the VM
performance.<br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 02/11/2016 03:13 PM, Bill James
wrote:<br class=3D"">
</div>
<blockquote cite=3D"mid:56BD1589.8090506@j2.com"
type=3D"cite" clas=
s=3D"">xml
attached. <br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/11/2016 12:28 PM, Nir Soffer wrote: <br class=3D"">
<blockquote type=3D"cite" class=3D"">On Thu, Feb 11,
2016 at 8:27=
PM, Bill
James <a moz-do-not-send=3D"true"
class=3D"moz-txt-link-rfc2396=
E"
href=3D"mailto:bill.james@j2.com"><bill.james@j2.com></a>
wrote: <br class=3D"">
<blockquote type=3D"cite" class=3D"">thank you for
the reply. <=
br class=3D"">
<br class=3D"">
We setup gluster using the names associated with NIC 2
IP. <br class=3D"">
Brick1:
<a href=3D"http://ovirt1-ks.test.j2noc.com"
class=3D"">ovirt1=
-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1 <br class=3D"">
Brick2:
<a href=3D"http://ovirt2-ks.test.j2noc.com"
class=3D"">ovirt2=
-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1 <br class=3D"">
Brick3:
<a href=3D"http://ovirt3-ks.test.j2noc.com"
class=3D"">ovirt3=
-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1 <br class=3D"">
<br class=3D"">
That's NIC 2's IP. <br class=3D"">
Using 'iftop -i eno2 -L 5 -t' : <br class=3D"">
<br class=3D"">
dd if=3D/dev/zero of=3D/root/testfile bs=3D1M count=3D1000
oflag=3Ddirect <br class=3D"">
1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s <br cl=
ass=3D"">
</blockquote>
Can you share the xml of this vm? You can find it in vdsm
log, <br class=3D"">
at the time you start the vm. <br class=3D"">
<br class=3D"">
Or you can do (on the host): <br class=3D"">
<br class=3D"">
# virsh <br class=3D"">
virsh # list <br class=3D"">
(username: vdsm@ovirt password: shibboleth) <br class=3D"">
virsh # dumpxml vm-id <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">Peak rate
(sent/received/total): &nb=
sp; =
281Mb 5.36Mb <br class=3D"">
282Mb <br class=3D"">
Cumulative
(sent/received/total): &nbs=
p; &=
nbsp;
1.96GB 14.6MB <br class=3D"">
1.97GB <br class=3D"">
<br class=3D"">
gluster volume info gv1: <br class=3D"">
Options Reconfigured: <br class=3D"">
performance.write-behind-window-size: 4MB <br class=3D=
"">
performance.readdir-ahead: on <br class=3D"">
performance.cache-size: 1GB <br class=3D"">
performance.write-behind: off <br class=3D"">
<br class=3D"">
performance.write-behind: off didn't help. <br
class=3D"">
Neither did any other changes I've tried. <br
class=3D"">
<br class=3D"">
<br class=3D"">
There is no VM traffic on this VM right now except my
test. <br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/10/2016 11:55 PM, Nir Soffer wrote: <br class=3D"">
<blockquote type=3D"cite" class=3D"">On Thu, Feb
11, 2016 at =
2:42 AM,
Ravishankar N <a moz-do-not-send=3D"true"
class=3D"moz-txt-=
link-rfc2396E"
href=3D"mailto:ravishankar@redhat.com"><ravishankar@redha=
t.com></a>
<br class=3D"">
wrote: <br class=3D"">
<blockquote type=3D"cite"
class=3D"">+gluster-users <br cla=
ss=3D"">
<br class=3D"">
Does disabling 'performance.write-behind' give a
better throughput? <br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/10/2016 11:06 PM, Bill James wrote: <br
class=3D"">
<blockquote type=3D"cite" class=3D"">I'm
setting up a ovi=
rt cluster
using glusterfs and noticing not stellar <br
class=3D"">
performance. <br class=3D"">
Maybe my setup could use some adjustments? <br class=3D=
"">
<br class=3D"">
3 hardware nodes running centos7.2, glusterfs
3.7.6.1, ovirt 3.6.2.6-1. <br class=3D"">
Each node has 8 spindles configured in 1 array which
is split using LVM <br class=3D"">
with one logical volume for system and one for
gluster. <br class=3D"">
They each have 4 NICs, <br class=3D"">
NIC1 =3D ovirtmgmt <br
class=3D"">
NIC2 =3D gluster (1GbE) <br
class=3D=
"">
</blockquote>
</blockquote>
How do you ensure that gluster trafic is using this nic?
<br class=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite"
class=3D""> NIC3 =
=3D VM traffic <br class=3D"">
</blockquote>
</blockquote>
How do you ensure that vm trafic is using this nic? <br cla=
ss=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite" class=3D"">I tried
with default=
glusterfs
settings <br class=3D"">
</blockquote>
</blockquote>
And did you find any difference? <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite" class=3D"">and also
with: <br c=
lass=3D"">
performance.cache-size: 1GB <br class=3D"">
performance.readdir-ahead: on <br class=3D"">
performance.write-behind-window-size: 4MB <br class=3D"=
">
<br class=3D"">
[root@ovirt3 test scripts]# gluster volume info gv1
<br class=3D"">
<br class=3D"">
Volume Name: gv1 <br class=3D"">
Type: Replicate <br class=3D"">
Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 <br cla=
ss=3D"">
Status: Started <br class=3D"">
Number of Bricks: 1 x 3 =3D 3 <br class=3D"">
Transport-type: tcp <br class=3D"">
Bricks: <br class=3D"">
Brick1:
<a href=3D"http://ovirt1-ks.test.j2noc.com"
class=3D"">=
ovirt1-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1 <br
class=3D"">
Brick2:
<a href=3D"http://ovirt2-ks.test.j2noc.com"
class=3D"">=
ovirt2-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1 <br
class=3D"">
Brick3:
<a href=3D"http://ovirt3-ks.test.j2noc.com"
class=3D"">=
ovirt3-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1 <br
class=3D"">
Options Reconfigured: <br class=3D"">
performance.cache-size: 1GB <br class=3D"">
performance.readdir-ahead: on <br class=3D"">
performance.write-behind-window-size: 4MB <br class=3D"=
">
<br class=3D"">
<br class=3D"">
Using simple dd test on VM in ovirt: <br class=3D"">
dd if=3D/dev/zero of=3D/root/testfil=
e bs=3D1G count=3D1
oflag=3Ddirect <br class=3D"">
</blockquote>
</blockquote>
block size of 1G?! <br class=3D"">
<br class=3D"">
Try 1M (our default for storage operations) <br
class=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite"
class=3D""> &n=
bsp; 1073741824 bytes (1.1
GB) copied, 65.9337 s, 16.3 MB/s <br class=3D"">
<br class=3D"">
Another VM not in ovirt using nfs: <br class=3D"">
dd if=3D/dev/zero
of=3D/root/t=
estfile bs=3D1G count=3D1
oflag=3Ddirect <br class=3D"">
1073741824 b=
ytes (1.1 GB) copied, 27.0079 s,
39.8 MB/s <br class=3D"">
<br class=3D"">
<br class=3D"">
Is that expected or is there a better way to set it
up to get better <br class=3D"">
performance? <br class=3D"">
</blockquote>
</blockquote>
Adding Niels for advice. <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">
<blockquote type=3D"cite" class=3D"">This
email, its cont=
ents and
.... <br class=3D"">
</blockquote>
</blockquote>
Please avoid this, this is a public mailing list,
everything you write <br class=3D"">
here is public. <br class=3D"">
<br class=3D"">
Nir <br class=3D"">
</blockquote>
I'll have to look into how to remove this sig for this
mailing list.... <br class=3D"">
<br class=3D"">
Cloud Services for Business <a moz-do-not-send=3D"true" class=
=3D"moz-txt-link-abbreviated"
href=3D"http://www.j2.com/">www.j2.com</a> <b=
r class=3D"">
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe |
Onebox <br class=3D"">
<br class=3D"">
<br class=3D"">
This email, its contents and attachments contain
information from j2 Global, <br class=3D"">
Inc. and/or its affiliates which may be privileged,
confidential or <br class=3D"">
otherwise protected from disclosure. The information is
intended to be for <br class=3D"">
the addressee(s) only. If you are not an addressee, any
disclosure, copy, <br class=3D"">
distribution, or use of the contents of this message is
prohibited. If you <br class=3D"">
have received this email in error please notify the sender
by reply e-mail <br class=3D"">
and delete the original message and any copies. (c) 2015
j2 Global, Inc. All <br class=3D"">
rights reserved. eFax, eVoice, Campaigner, FuseMail,
KeepItSafe, and Onebox <br class=3D"">
are registered trademarks of j2 Global, Inc. and its
affiliates. <br class=3D"">
</blockquote>
</blockquote>
<br class=3D"">
<br class=3D"">
<fieldset class=3D"mimeAttachmentHeader"></fieldset>
<br class=3D"">
<pre wrap=3D""
class=3D"">_______________________________________=
________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated"
href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext"
href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ma...
info/users</a>
</pre>
</blockquote>
<br class=3D""><p class=3D""><a
moz-do-not-send=3D"true" href=3D"ht=
tp://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-referra...
utm_campaign=3Demployeeemail" class=3D""><span
style=3D"color:windowtext; t=
ext-decoration:none" class=3D""><img
moz-do-not-send=3D"true" src=3D"http:/=
/home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Footer.jpg" alt=3D"ww=
w.j2.com" height=3D"46" width=3D"391" border=3D"0"
class=3D""></span></a></=
p><p class=3D""><span
style=3D"font-size:8.0pt;font-family:"Arial"=
;,"sans-serif";color:gray" class=3D"">This
email, its contents and attachments contain information from
<a moz-do-not-send=3D"true"
href=3D"http://www.j2.com/?utm_sour=
ce=3Dj2global&utm_medium=3Dxsell-referral&utm_campaign=3Demployemai=
l" class=3D"">j2
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of
this message is prohibited. If you have received this email
in error please notify the sender by reply e-mail and delete
the original message and any copies. =A9 2015 <a moz-do-not-sen=
d=3D"true" href=3D"http://www.j2.com/" class=3D"">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send=3D"t=
rue" href=3D"http://www.efax.com/" class=3D"">eFax
=AE</a>,
<a moz-do-not-send=3D"true"
href=3D"http://www.evoice.com/" cla=
ss=3D"">eVoice
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"http://www.campa=
igner.com/" class=3D"">Campaigner =AE</a>, <a
moz-do-not-send=3D"true" href=
=3D"http://www.fusemail.com/" class=3D"">FuseMail
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"http://www.keepi=
tsafe.com/" class=3D"">KeepItSafe =AE</a> and <a
moz-do-not-send=3D"true" h=
ref=3D"http://www.onebox.com/" class=3D"">Onebox
=AE</a> are r egistered trademarks of <a
moz-do-not-send=3D"t=
rue" href=3D"http://www.j2.com/" class=3D"">j2
Global, Inc</a>. and its affiliates.</span></p>
<br class=3D"">
<fieldset class=3D"mimeAttachmentHeader"></fieldset>
<br class=3D"">
<pre wrap=3D""
class=3D"">_________________________________________=
______
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated"
href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext"
href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ma...
info/users</a>
</pre>
</blockquote>
<br class=3D""><p class=3D""><a
moz-do-not-send=3D"true" href=3D"http=
://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-referral&ut=
m_campaign=3Demployeeemail" class=3D""><span
style=3D"color:windowtext;
text-decoration:none" class=3D""><img
moz-do-not-send=3D"true" =
src=3D"http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_F...
pg" alt=3D"www.j2.com" height=3D"46" width=3D"391"
border=3D"0" class=3D"">=
</span></a></p><p class=3D""><span
style=3D"font-size:8.0pt;font-family:&qu=
ot;Arial","sans-serif";color:gray"
class=3D"">This
email, its contents and attachments contain information from <a m=
oz-do-not-send=3D"true"
href=3D"http://www.j2.com/?utm_source=3Dj2global&am=
p;utm_medium=3Dxsell-referral&utm_campaign=3Demployemail"
class=3D"">j2
Global, Inc</a>. and/or its affiliates which may be
privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any
disclosure, copy, distribution, or use of the contents of this
message is prohibited. If you have received this email in
error please notify the sender by reply e-mail and delete the
original message and any copies. =A9 2015 <a moz-do-not-send=3D"t=
rue" href=3D"http://www.j2.com/" class=3D"">j2 Global,
Inc</a>. All rights reserved. <a moz-do-not-send=3D"true"
href=
=3D"http://www.efax.com/" class=3D"">eFax =AE</a>, <a
moz-do-not-send=3D"tr=
ue" href=3D"http://www.evoice.com/" class=3D"">eVoice
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"http://www.campaig=
ner.com/" class=3D"">Campaigner =AE</a>, <a
moz-do-not-send=3D"true" href=
=3D"http://www.fusemail.com/" class=3D"">FuseMail
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"http://www.keepits=
afe.com/" class=3D"">KeepItSafe =AE</a> and <a
moz-do-not-send=3D"true" hre=
f=3D"http://www.onebox.com/" class=3D"">Onebox
=AE</a> are r egistered trademarks of <a
moz-do-not-send=3D"tru=
e" href=3D"http://www.j2.com/" class=3D"">j2 Global,
Inc</a>. and its affiliates.</span></p>
<br class=3D"">
<fieldset class=3D"mimeAttachmentHeader"></fieldset>
<br class=3D"">
<pre wrap=3D""
class=3D"">___________________________________________=
____
Users mailing list
<a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:Users@ovirt.org">Users=
@ovirt.org</a>
<a class=3D"moz-txt-link-freetext"
href=3D"http://lists.ovirt.org/mailman/l=
istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</...
</pre>
</blockquote>
<br class=3D"">
<br class=3D"">
<font face=3D"Verdana,Arial,Helvetica,Trebuchet MS" size=3D"1"
class=3D"">
<br class=3D"">--=20
<br class=3D"">This message is subject to the CSIR's copyright terms
and co=
nditions, e-mail legal notice, and implemented Open Document Format (ODF) s=
tandard.
<br class=3D"">The full disclaimer details can be found at <a
href=3D"http:=
//www.csir.co.za/disclaimer.html"
class=3D"">http://www.csir.co.za/disclaim=
er.html</a>.
<p class=3D"">
<br class=3D"">This message has been scanned for viruses and dangerous
cont=
ent by <a href=3D"http://www.mailscanner.info/" class=3D""><b
class=3D"">Ma=
ilScanner</b></a>,=20
<br class=3D"">and is believed to be clean.
</p></font><p class=3D""><font
face=3D"Verdana,Arial,Helvetica,Trebuchet MS=
" size=3D"1" class=3D"">
<br class=3D"">Please consider the environment before printing this
email.
</font>
</p></div>
_______________________________________________<br class=3D"">Users
mailing=
list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
class=3D"">Users@ovi=
rt.org</a><br
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br c=
lass=3D""></div></blockquote></div><br
class=3D""></div></div></div><font f=
ace=3D"Verdana,Arial,Helvetica,Trebuchet MS" size=3D"1">
<br />--=20
<br />This message is subject to the CSIR's copyright terms and conditions,=
e-mail legal notice, and implemented Open Document Format (ODF) standard.
<br />The full disclaimer details can be found at <a
href=3D"http://www.csi=
r.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a>.
<p>
<br />This message has been scanned for viruses and dangerous content by <a=
href=3D"http://www.mailscanner.info/"><b>MailScanner</b></a>,=20
<br />and is believed to be clean.
<p>
<br />Please consider the environment before printing this email.
</font>
</body></html>
--Apple-Mail=_F4D45E80-CC6B-492F-BC3F-6304354C342D--
11:56 a.m.
This is a multi-part message in MIME format.
--------------010607050604070801030201
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
On 04/06/2016 02:08 PM, Roderick Mooi wrote:
Hi Ravi and colleagues
(apologies for hijacking this thread but Im not sure where else to
report this (and it is related).)
With gluster 3.7.10, running
#gluster volume set <volname> group virt
fails with:
volume set: failed: option : eager-lock does not exist
Did you mean eager-lock?
I had to remove the eager-lock setting from
/var/lib/glusterd/groups/virt to get this to work. It seems like
setting eager-lock has been removed from latest gluster. Is this
correct? Either way, is there anything else I should do?
It is not removed. Can you try 'gluster volume set volname
cluster.eager-lock enable`?
I think the disperse (EC) translator introduced a `disperse.eager-lock`
which is why you would need to mention entire volume option name to
avoid ambiguity.
We probably need to fix the virt profile setting to include the entire
name. By the way 'gluster volume set help` should give you the list of
all options.
-Ravi
Cheers,
Roderick
> On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishankar(a)redhat.com
> <mailto:ravishankar@redhat.com>> wrote:
>
> Hi Bill,
> Can you enable virt-profile setting for your volume and see if that
> helps? You need to enable this optimization when you create the
> volume using ovrit, or use the following command for an existing volume:
>
> #gluster volume set <volname> group virt
>
> -Ravi
>
>
> On 02/12/2016 05:22 AM, Bill James wrote:
>> My apologies, I'm showing how much of a noob I am.
>> Ignore last direct to gluster numbers, as that wasn't really glusterfs.
>>
>>
>> [root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com
>> <http://ovirt2-ks.test.j2noc.com>:/gv1 /mnt/tmp/
>> [root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
>> bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
>>
>> That's more how I expected, it is pointing to glusterfs performance.
>>
>>
>>
>> On 02/11/2016 03:27 PM, Bill James wrote:
>>> don't know if it helps, but I ran a few more tests, all from the
>>> same hardware node.
>>>
>>> The VM:
>>> [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
>>> count=1000 oflag=direct
>>> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
>>>
>>> Writing directly to gluster volume:
>>> [root@ovirt2 test ~]# time dd if=/dev/zero
>>> of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
>>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
>>>
>>>
>>> Writing to NFS volume:
>>> [root@ovirt2 test ~]# time dd if=/dev/zero
>>> of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct
>>> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
>>>
>>> NFS & Gluster are using the same interface. Tests were not run at
>>> same time.
>>>
>>> This would suggest my problem isn't glusterfs, but the VM performance.
>>>
>>>
>>>
>>> On 02/11/2016 03:13 PM, Bill James wrote:
>>>> xml attached.
>>>>
>>>>
>>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:
>>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com>
>>>>> wrote:
>>>>>> thank you for the reply.
>>>>>>
>>>>>> We setup gluster using the names associated with NIC 2 IP.
>>>>>> Brick1: ovirt1-ks.test.j2noc.com
>>>>>>
<http://ovirt1-ks.test.j2noc.com>:/gluster-store/brick1/gv1
>>>>>> Brick2: ovirt2-ks.test.j2noc.com
>>>>>>
<http://ovirt2-ks.test.j2noc.com>:/gluster-store/brick1/gv1
>>>>>> Brick3: ovirt3-ks.test.j2noc.com
>>>>>>
<http://ovirt3-ks.test.j2noc.com>:/gluster-store/brick1/gv1
>>>>>>
>>>>>> That's NIC 2's IP.
>>>>>> Using 'iftop -i eno2 -L 5 -t' :
>>>>>>
>>>>>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>>>>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
>>>>> Can you share the xml of this vm? You can find it in vdsm log,
>>>>> at the time you start the vm.
>>>>>
>>>>> Or you can do (on the host):
>>>>>
>>>>> # virsh
>>>>> virsh # list
>>>>> (username: vdsm@ovirt password: shibboleth)
>>>>> virsh # dumpxml vm-id
>>>>>
>>>>>> Peak rate (sent/received/total): 281Mb 5.36Mb
>>>>>> 282Mb
>>>>>> Cumulative (sent/received/total): 1.96GB 14.6MB
>>>>>> 1.97GB
>>>>>>
>>>>>> gluster volume info gv1:
>>>>>> Options Reconfigured:
>>>>>> performance.write-behind-window-size: 4MB
>>>>>> performance.readdir-ahead: on
>>>>>> performance.cache-size: 1GB
>>>>>> performance.write-behind: off
>>>>>>
>>>>>> performance.write-behind: off didn't help.
>>>>>> Neither did any other changes I've tried.
>>>>>>
>>>>>>
>>>>>> There is no VM traffic on this VM right now except my test.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
>>>>>>> <ravishankar(a)redhat.com>
>>>>>>> wrote:
>>>>>>>> +gluster-users
>>>>>>>>
>>>>>>>> Does disabling 'performance.write-behind' give a
better
>>>>>>>> throughput?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>>>>>> I'm setting up a ovirt cluster using glusterfs
and noticing
>>>>>>>>> not stellar
>>>>>>>>> performance.
>>>>>>>>> Maybe my setup could use some adjustments?
>>>>>>>>>
>>>>>>>>> 3 hardware nodes running centos7.2, glusterfs
3.7.6.1, ovirt
>>>>>>>>> 3.6.2.6-1.
>>>>>>>>> Each node has 8 spindles configured in 1 array which
is split
>>>>>>>>> using LVM
>>>>>>>>> with one logical volume for system and one for
gluster.
>>>>>>>>> They each have 4 NICs,
>>>>>>>>> NIC1 = ovirtmgmt
>>>>>>>>> NIC2 = gluster (1GbE)
>>>>>>> How do you ensure that gluster trafic is using this nic?
>>>>>>>
>>>>>>>>> NIC3 = VM traffic
>>>>>>> How do you ensure that vm trafic is using this nic?
>>>>>>>
>>>>>>>>> I tried with default glusterfs settings
>>>>>>> And did you find any difference?
>>>>>>>
>>>>>>>>> and also with:
>>>>>>>>> performance.cache-size: 1GB
>>>>>>>>> performance.readdir-ahead: on
>>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>>
>>>>>>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>>>>>>
>>>>>>>>> Volume Name: gv1
>>>>>>>>> Type: Replicate
>>>>>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>>>>>> Status: Started
>>>>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>>>>> Transport-type: tcp
>>>>>>>>> Bricks:
>>>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
>>>>>>>>>
<http://ovirt1-ks.test.j2noc.com>:/gluster-store/brick1/gv1
>>>>>>>>> Brick2: ovirt2-ks.test.j2noc.com
>>>>>>>>>
<http://ovirt2-ks.test.j2noc.com>:/gluster-store/brick1/gv1
>>>>>>>>> Brick3: ovirt3-ks.test.j2noc.com
>>>>>>>>>
<http://ovirt3-ks.test.j2noc.com>:/gluster-store/brick1/gv1
>>>>>>>>> Options Reconfigured:
>>>>>>>>> performance.cache-size: 1GB
>>>>>>>>> performance.readdir-ahead: on
>>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Using simple dd test on VM in ovirt:
>>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>>> block size of 1G?!
>>>>>>>
>>>>>>> Try 1M (our default for storage operations)
>>>>>>>
>>>>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3
MB/s
>>>>>>>>>
>>>>>>>>> Another VM not in ovirt using nfs:
>>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G count=1
>>>>>>>>> oflag=direct
>>>>>>>>> 1073741824 bytes (1.1 GB) copied, 27.0079 s,
39.8 MB/s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Is that expected or is there a better way to set it
up to get
>>>>>>>>> better
>>>>>>>>> performance?
>>>>>>> Adding Niels for advice.
>>>>>>>
>>>>>>>>> This email, its contents and ....
>>>>>>> Please avoid this, this is a public mailing list, everything
>>>>>>> you write
>>>>>>> here is public.
>>>>>>>
>>>>>>> Nir
>>>>>> I'll have to look into how to remove this sig for this
mailing
>>>>>> list....
>>>>>>
>>>>>> Cloud Services for Business www.j2.com
>>>>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>>>>>
>>>>>>
>>>>>> This email, its contents and attachments contain information
>>>>>> from j2 Global,
>>>>>> Inc. and/or its affiliates which may be privileged, confidential
or
>>>>>> otherwise protected from disclosure. The information is intended
>>>>>> to be for
>>>>>> the addressee(s) only. If you are not an addressee, any
>>>>>> disclosure, copy,
>>>>>> distribution, or use of the contents of this message is
>>>>>> prohibited. If you
>>>>>> have received this email in error please notify the sender by
>>>>>> reply e-mail
>>>>>> and delete the original message and any copies. (c) 2015 j2
>>>>>> Global, Inc. All
>>>>>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe,
>>>>>> and Onebox
>>>>>> are registered trademarks of j2 Global, Inc. and its affiliates.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>> www.j2.com
>>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>>
>>> This email, its contents and attachments contain information from
>>> j2 Global, Inc
>>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
>>> and/or its affiliates which may be privileged, confidential or
>>> otherwise protected from disclosure. The information is intended to
>>> be for the addressee(s) only. If you are not an addressee, any
>>> disclosure, copy, distribution, or use of the contents of this
>>> message is prohibited. If you have received this email in error
>>> please notify the sender by reply e-mail and delete the original
>>> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;.
>>> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ®
>>> <http://www.evoice.com/>;, Campaigner ®
>>> <http://www.campaigner.com/>;, FuseMail ®
>>> <http://www.fusemail.com/>;, KeepItSafe ®
>>> <http://www.keepitsafe.com/> and Onebox ®
<http://www.onebox.com/>
>>> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
>>> and its affiliates.
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>> www.j2.com
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>
>> This email, its contents and attachments contain information from j2
>> Global, Inc
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
>> and/or its affiliates which may be privileged, confidential or
>> otherwise protected from disclosure. The information is intended to
>> be for the addressee(s) only. If you are not an addressee, any
>> disclosure, copy, distribution, or use of the contents of this
>> message is prohibited. If you have received this email in error
>> please notify the sender by reply e-mail and delete the original
>> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;.
>> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ®
>> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,
>> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ®
>> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/>
>> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
>> and its affiliates.
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> --
> This message is subject to the CSIR's copyright terms and conditions,
> e-mail legal notice, and implemented Open Document Format (ODF)
> standard.
> The full disclaimer details can be found at
> http://www.csir.co.za/disclaimer.html.
>
>
> This message has been scanned for viruses and dangerous content by
> *MailScanner* <http://www.mailscanner.info/>;,
> and is believed to be clean.
>
>
> Please consider the environment before printing this email.
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
--
This message is subject to the CSIR's copyright terms and conditions,
e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at
http://www.csir.co.za/disclaimer.html.
This message has been scanned for viruses and dangerous content by
*MailScanner* <http://www.mailscanner.info/>;,
and is believed to be clean.
Please consider the environment before printing this email.
--------------010607050604070801030201
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 04/06/2016 02:08 PM, Roderick Mooi
wrote:<br>
</div>
<blockquote
cite="mid:90A2411F-0F73-437D-899B-6098A5C7A1B6@csir.co.za"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hi Ravi and colleagues
<div class=""><br class="">
</div>
<div class="">(apologies for hijacking this thread but Im not
sure where else to report this (and it is related).)
<div class=""><br class="">
</div>
<div class="">With gluster 3.7.10, running</div>
<div class="">#gluster volume set <volname> group
virt</div>
<div class="">fails with:</div>
<div class="">volume set: failed: option : eager-lock does not
exist</div>
<div class="">Did you mean eager-lock?</div>
<div class=""><br class="">
</div>
<div class="">I had to remove the eager-lock setting from
/var/lib/glusterd/groups/virt to get this to work. It seems
like setting eager-lock has been removed from latest gluster.
Is this correct? Either way, is there anything else I should
do?</div>
</div>
</blockquote>
<br>
It is not removed. Can you try 'gluster volume set volname
cluster.eager-lock enable`?<br>
I think the disperse (EC) translator introduced a
`disperse.eager-lock` which is why you would need to mention entire
volume option name to avoid ambiguity.<br>
We probably need to fix the virt profile setting to include the
entire name. By the way 'gluster volume set help` should give you
the list of all options.<br>
<br>
-Ravi<br>
<br>
<blockquote
cite="mid:90A2411F-0F73-437D-899B-6098A5C7A1B6@csir.co.za"
type="cite">
<div class="">
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">Roderick<br class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 12 Feb 2016, at 6:18 AM, Ravishankar N
<<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
class="">ravishankar(a)redhat.com</a>&gt;
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF"
class="">
<div class="moz-cite-prefix">Hi Bill,<br
class="">
Can you enable virt-profile setting for your
volume and see if that helps? You need to enable
this optimization when you create the volume using
ovrit, or use the following command for an
existing volume:<br class="">
<tt class=""><br class="">
#gluster volume set <volname> group
virt</tt><br
class="">
<br class="">
-Ravi<br class="">
<br class="">
<br class="">
On 02/12/2016 05:22 AM, Bill James wrote:<br
class="">
</div>
<blockquote cite="mid:56BD1EB9.5000802@j2.com"
type="cite" class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
My apologies, I'm showing how much of a noob I am.<br
class="">
Ignore last direct to gluster numbers, as that
wasn't really glusterfs.<br class="">
<br class="">
<br class="">
[root@ovirt2 test ~]# mount -t glusterfs <a
moz-do-not-send="true"
href="http://ovirt2-ks.test.j2noc.com"
class="">ovirt2-ks.test.j2noc.com</a>:/gv1
/mnt/tmp/<br class="">
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/mnt/tmp/testfile2 bs=1M count=1000
oflag=direct<br class="">
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9
MB/s<br class="">
<br class="">
That's more how I expected, it is pointing to
glusterfs performance.<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 02/11/2016 03:27
PM, Bill James wrote:<br class="">
</div>
<blockquote cite="mid:56BD18D7.8080909@j2.com"
type="cite" class="">
<meta http-equiv="Content-Type"
content="text/html; charset=windows-1252"
class="">
don't know if it helps, but I ran a few more
tests, all from the same hardware node.<br
class="">
<br class="">
The VM:<br class="">
[root@billjov1 ~]# time dd if=/dev/zero
of=/root/testfile bs=1M count=1000 oflag=direct<br
class="">
1048576000 bytes (1.0 GB) copied, 62.5535 s,
16.8 MB/s<br class="">
<br class="">
Writing directly to gluster volume:<br class="">
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/gluster-store/brick1/gv1/testfile bs=1M
count=1000 oflag=direct<br class="">
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106
MB/s<br class="">
<br class="">
<br class="">
Writing to NFS volume:<br class="">
[root@ovirt2 test ~]# time dd if=/dev/zero
of=/mnt/storage/qa/testfile bs=1M count=1000
oflag=direct<br class="">
1048576000 bytes (1.0 GB) copied, 10.5776 s,
99.1 MB/s<br class="">
<br class="">
NFS & Gluster are using the same interface.
Tests were not run at same time.<br class="">
<br class="">
This would suggest my problem isn't glusterfs,
but the VM performance.<br class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On 02/11/2016 03:13
PM, Bill James wrote:<br class="">
</div>
<blockquote cite="mid:56BD1589.8090506@j2.com"
type="cite" class="">xml attached.
<br
class="">
<br class="">
<br class="">
On 02/11/2016 12:28 PM, Nir Soffer wrote: <br
class="">
<blockquote type="cite" class="">On
Thu, Feb
11, 2016 at 8:27 PM, Bill James <a
moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><a
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a></a>
wrote: <br class="">
<blockquote type="cite"
class="">thank you
for the reply. <br class="">
<br class="">
We setup gluster using the names
associated with NIC 2 IP. <br class="">
Brick1: <a moz-do-not-send="true"
href="http://ovirt1-ks.test.j2noc.com"
class="">ovirt1-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick2: <a moz-do-not-send="true"
href="http://ovirt2-ks.test.j2noc.com"
class="">ovirt2-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick3: <a moz-do-not-send="true"
href="http://ovirt3-ks.test.j2noc.com"
class="">ovirt3-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
<br class="">
That's NIC 2's IP. <br class="">
Using 'iftop -i eno2 -L 5 -t' : <br
class="">
<br class="">
dd if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct <br class="">
1048576000 bytes (1.0 GB) copied, 68.0714
s, 15.4 MB/s <br class="">
</blockquote>
Can you share the xml of this vm? You can
find it in vdsm log, <br class="">
at the time you start the vm. <br class="">
<br class="">
Or you can do (on the host): <br class="">
<br class="">
# virsh <br class="">
virsh # list <br class="">
(username: vdsm@ovirt password: shibboleth)
<br class="">
virsh # dumpxml vm-id <br class="">
<br class="">
<blockquote type="cite"
class="">Peak rate
(sent/received/total):
281Mb 5.36Mb <br class="">
282Mb <br class="">
Cumulative
(sent/received/total):
1.96GB 14.6MB <br class="">
1.97GB <br class="">
<br class="">
gluster volume info gv1: <br class="">
Options Reconfigured: <br class="">
performance.write-behind-window-size:
4MB <br class="">
performance.readdir-ahead: on <br
class="">
performance.cache-size: 1GB <br
class="">
performance.write-behind: off <br
class="">
<br class="">
performance.write-behind: off didn't help.
<br class="">
Neither did any other changes I've tried.
<br class="">
<br class="">
<br class="">
There is no VM traffic on this VM right
now except my test. <br class="">
<br class="">
<br class="">
<br class="">
On 02/10/2016 11:55 PM, Nir Soffer wrote:
<br class="">
<blockquote type="cite"
class="">On Thu,
Feb 11, 2016 at 2:42 AM, Ravishankar N <a
moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><a
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a></a>
<br class="">
wrote: <br class="">
<blockquote type="cite"
class="">+gluster-users
<br class="">
<br class="">
Does disabling
'performance.write-behind' give a
better throughput? <br class="">
<br class="">
<br class="">
<br class="">
On 02/10/2016 11:06 PM, Bill James
wrote: <br class="">
<blockquote type="cite"
class="">I'm
setting up a ovirt cluster using
glusterfs and noticing not stellar <br
class="">
performance. <br class="">
Maybe my setup could use some
adjustments? <br class="">
<br class="">
3 hardware nodes running centos7.2,
glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
<br class="">
Each node has 8 spindles configured
in 1 array which is split using LVM
<br class="">
with one logical volume for system
and one for gluster. <br class="">
They each have 4 NICs, <br class="">
NIC1 = ovirtmgmt <br class="">
NIC2 = gluster (1GbE) <br
class="">
</blockquote>
</blockquote>
How do you ensure that gluster trafic is
using this nic? <br class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">
NIC3 = VM traffic <br class="">
</blockquote>
</blockquote>
How do you ensure that vm trafic is
using this nic? <br class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">I
tried with default glusterfs
settings <br class="">
</blockquote>
</blockquote>
And did you find any difference? <br
class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">and
also with: <br class="">
performance.cache-size: 1GB <br
class="">
performance.readdir-ahead: on <br
class="">
performance.write-behind-window-size:
4MB <br class="">
<br class="">
[root@ovirt3 test scripts]# gluster
volume info gv1 <br class="">
<br class="">
Volume Name: gv1 <br class="">
Type: Replicate <br class="">
Volume ID:
71afc35b-09d7-4384-ab22-57d032a0f1a2
<br class="">
Status: Started <br class="">
Number of Bricks: 1 x 3 = 3 <br
class="">
Transport-type: tcp <br class="">
Bricks: <br class="">
Brick1: <a moz-do-not-send="true"
href="http://ovirt1-ks.test.j2noc.com"
class="">ovirt1-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick2: <a moz-do-not-send="true"
href="http://ovirt2-ks.test.j2noc.com"
class="">ovirt2-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick3: <a moz-do-not-send="true"
href="http://ovirt3-ks.test.j2noc.com"
class="">ovirt3-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Options Reconfigured: <br class="">
performance.cache-size: 1GB <br
class="">
performance.readdir-ahead: on <br
class="">
performance.write-behind-window-size:
4MB <br class="">
<br class="">
<br class="">
Using simple dd test on VM in ovirt:
<br class="">
dd if=/dev/zero
of=/root/testfile bs=1G count=1
oflag=direct <br class="">
</blockquote>
</blockquote>
block size of 1G?! <br class="">
<br class="">
Try 1M (our default for storage
operations) <br class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">
1073741824 bytes (1.1 GB) copied,
65.9337 s, 16.3 MB/s <br class="">
<br class="">
Another VM not in ovirt using nfs: <br
class="">
dd if=/dev/zero
of=/root/testfile bs=1G count=1
oflag=direct <br class="">
1073741824 bytes (1.1 GB)
copied, 27.0079 s, 39.8 MB/s <br
class="">
<br class="">
<br class="">
Is that expected or is there a
better way to set it up to get
better <br class="">
performance? <br class="">
</blockquote>
</blockquote>
Adding Niels for advice. <br class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">This
email, its contents and .... <br
class="">
</blockquote>
</blockquote>
Please avoid this, this is a public
mailing list, everything you write <br
class="">
here is public. <br class="">
<br class="">
Nir <br class="">
</blockquote>
I'll have to look into how to remove this
sig for this mailing list.... <br
class="">
<br class="">
Cloud Services for Business <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="http://www.j2.com/"><a
class="moz-txt-link-abbreviated"
href="http://www.j2.com">www.j2.com</a></a>
<br class="">
j2 | eFax | eVoice | FuseMail | Campaigner
| KeepItSafe | Onebox <br class="">
<br class="">
<br class="">
This email, its contents and attachments
contain information from j2 Global, <br
class="">
Inc. and/or its affiliates which may be
privileged, confidential or <br class="">
otherwise protected from disclosure. The
information is intended to be for <br
class="">
the addressee(s) only. If you are not an
addressee, any disclosure, copy, <br
class="">
distribution, or use of the contents of
this message is prohibited. If you <br
class="">
have received this email in error please
notify the sender by reply e-mail <br
class="">
and delete the original message and any
copies. (c) 2015 j2 Global, Inc. All <br
class="">
rights reserved. eFax, eVoice, Campaigner,
FuseMail, KeepItSafe, and Onebox <br
class="">
are registered trademarks of j2 Global,
Inc. and its affiliates. <br class="">
</blockquote>
</blockquote>
<br class="">
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
<p class=""><a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class=""><span style="color:windowtext;
text-decoration:none" class=""><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" class=""
height="46"
width="391"
border="0"></span></a></p>
<p class=""><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray"
class="">This email, its contents and
attachments contain information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class="">j2 Global, Inc</a>. and/or its
affiliates which may be privileged,
confidential or otherwise protected from
disclosure. The information is intended to
be for the addressee(s) only. If you are not
an addressee, any disclosure, copy,
distribution, or use of the contents of this
message is prohibited. If you have received
this email in error please notify the sender
by reply e-mail and delete the original
message and any copies. © 2015 <a
moz-do-not-send="true"
href="http://www.j2.com/"
class="">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/"
class="">eFax
®</a>, <a moz-do-not-send="true"
href="http://www.evoice.com/"
class="">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/"
class="">Campaigner
®</a>, <a moz-do-not-send="true"
href="http://www.fusemail.com/"
class="">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/"
class="">KeepItSafe
®</a> and <a moz-do-not-send="true"
href="http://www.onebox.com/"
class="">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true"
href="http://www.j2.com/"
class="">j2
Global, Inc</a>. and its
affiliates.</span></p>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
<p class=""><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class=""><span style="color:windowtext;
text-decoration:none" class=""><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" class=""
height="46"
width="391"
border="0"></span></a></p>
<p class=""><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray"
class="">This email, its contents and
attachments contain information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class="">j2 Global, Inc</a>. and/or its
affiliates which may be privileged,
confidential or otherwise protected from
disclosure. The information is intended to be
for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution,
or use of the contents of this message is
prohibited. If you have received this email in
error please notify the sender by reply e-mail
and delete the original message and any
copies. © 2015 <a moz-do-not-send="true"
href="http://www.j2.com/" class="">j2
Global, Inc</a>. All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/"
class="">eFax ®</a>,
<a moz-do-not-send="true"
href="http://www.evoice.com/"
class="">eVoice
®</a>, <a moz-do-not-send="true"
href="http://www.campaigner.com/"
class="">Campaigner
®</a>, <a moz-do-not-send="true"
href="http://www.fusemail.com/"
class="">FuseMail
®</a>, <a moz-do-not-send="true"
href="http://www.keepitsafe.com/"
class="">KeepItSafe
®</a> and <a moz-do-not-send="true"
href="http://www.onebox.com/"
class="">Onebox
®</a> are r egistered trademarks of <a
moz-do-not-send="true"
href="http://www.j2.com/" class="">j2
Global, Inc</a>. and its
affiliates.</span></p>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
<br class="">
<font class=""
face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1">
<br class="">
-- <br class="">
This message is subject to the CSIR's copyright
terms and conditions, e-mail legal notice, and
implemented Open Document Format (ODF) standard.
<br class="">
The full disclaimer details can be found at <a
moz-do-not-send="true"
href="http://www.csir.co.za/disclaimer.html"
class=""><a
class="moz-txt-link-freetext"
href="http://www.csir.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a></a>.
<p class="">
<br class="">
This message has been scanned for viruses and
dangerous content by <a moz-do-not-send="true"
href="http://www.mailscanner.info/"
class=""><b
class="">MailScanner</b></a>, <br
class="">
and is believed to be clean.
</p>
</font>
<p class=""><font class=""
face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1">
<br class="">
Please consider the environment before printing
this email.
</font>
</p>
</div>
_______________________________________________<br
class="">
Users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
class="">Users(a)ovirt.org</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<font face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1">
<br>
-- <br>
This message is subject to the CSIR's copyright terms and
conditions, e-mail legal notice, and implemented Open Document
Format (ODF) standard.
<br>
The full disclaimer details can be found at <a
moz-do-not-send="true"
href="http://www.csir.co.za/disclaimer.html"><a
class="moz-txt-link-freetext"
href="http://www.csir.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a></a>.
<p>
<br>
This message has been scanned for viruses and dangerous
content by <a moz-do-not-send="true"
href="http://www.mailscanner.info/"><b>MailScanner</b></a>,
<br>
and is believed to be clean.
</p>
</font>
<p><font face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1">
<br>
Please consider the environment before printing this email.
</font>
</p>
</blockquote>
<br>
<br>
</body>
</html>
--------------010607050604070801030201--
12:11 p.m.
--Apple-Mail=_2474E8FB-D4B8-41DB-B4A1-843303C4F112
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
Hi
It is not removed. Can you try 'gluster volume set volname
cluster.eager-=
lock enable`?
This works. BTW by default this setting is =93on=94. What=92s the differenc=
e between =93on=94 and =93enable=94?
Thanks for the clarification.
Regards,
Roderick
On 06 Apr 2016, at 10:56 AM, Ravishankar N
<ravishankar(a)redhat.com> wrote:
=20
On 04/06/2016 02:08 PM, Roderick Mooi wrote:
> Hi Ravi and colleagues
>=20
> (apologies for hijacking this thread but I=92m not sure where else to re=
port
this (and it is related).)
>=20
> With gluster 3.7.10, running
> #gluster volume set <volname> group virt
> fails with:
> volume set: failed: option : eager-lock does not exist
> Did you mean eager-lock?
>=20
> I had to remove the eager-lock setting from /var/lib/glusterd/groups/vir=
t to
get this to work. It seems like setting eager-lock has been removed fr=
om latest gluster. Is this correct? Either way, is there anything else I sh=
ould do?
>=20
It is not removed. Can you try 'gluster volume set volname
cluster.eager-=
lock enable`?
I think the disperse (EC) translator introduced a
`disperse.eager-lock` w=
hich is why you would need to mention entire volume option
name to avoid am=
biguity.
We probably need to fix the virt profile setting to include the
entire na=
me. By the way 'gluster volume set help` should give you the list of
all op=
tions.
=20
-Ravi
=20
>=20
> Cheers,
>=20
> Roderick
>=20
>> On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishankar(a)redhat.com <mail=
to:ravishankar@redhat.com>> wrote:
>>=20
>> Hi Bill,
>> Can you enable virt-profile setting for your volume and see if that hel=
ps? You need to enable this optimization when you create the volume using o=
vrit, or use the following command for an existing volume:
>>=20
>> #gluster volume set <volname> group virt
>>=20
>> -Ravi
>>=20
>>=20
>> On 02/12/2016 05:22 AM, Bill James wrote:
>>> My apologies, I'm showing how much of a noob I am.
>>> Ignore last direct to gluster numbers, as that wasn't really glusterfs.
>>>=20
>>>=20
>>> [root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com <htt=
p://ovirt2-ks.test.j2noc.com/>:/gv1 /mnt/tmp/
>>> [root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/mnt/tmp/testfile2 b=
s=3D1M count=3D1000 oflag=3Ddirect
>>> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
>>>=20
>>> That's more how I expected, it is pointing to glusterfs performance.
>>>=20
>>>=20
>>>=20
>>> On 02/11/2016 03:27 PM, Bill James wrote:
>>>> don't know if it helps, but I ran a few more tests, all from the
same=
hardware node.
>>>>=20
>>>> The VM:
>>>> [root@billjov1 ~]# time dd if=3D/dev/zero of=3D/root/testfile bs=3D1M=
count=3D1000 oflag=3Ddirect
>>>> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8
MB/s
>>>>=20
>>>> Writing directly to gluster volume:
>>>> [root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/gluster-store/bric=
k1/gv1/testfile bs=3D1M count=3D1000 oflag=3Ddirect
>>>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106
MB/s
>>>>=20
>>>>=20
>>>> Writing to NFS volume:
>>>> [root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/mnt/storage/qa/tes=
tfile bs=3D1M count=3D1000 oflag=3Ddirect
>>>> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1
MB/s
>>>>=20
>>>> NFS & Gluster are using the same interface. Tests were not run at
sam=
e time.
>>>>=20
>>>> This would suggest my problem isn't glusterfs, but the VM
performance.
>>>>=20
>>>>=20
>>>>=20
>>>> On 02/11/2016 03:13 PM, Bill James wrote:
>>>>> xml attached.=20
>>>>>=20
>>>>>=20
>>>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:=20
>>>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James
<mailto:bill.james@j2.=
com><bill.james(a)j2.com>
<mailto:bill.james@j2.com> wrote:=20
>>>>>>> thank you for the reply.=20
>>>>>>>=20
>>>>>>> We setup gluster using the names associated with NIC 2
IP.=20
>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
<http://ovirt1-ks.test.j2noc.co=
m/>:/gluster-store/brick1/gv1=20
>>>>>>> Brick2: ovirt2-ks.test.j2noc.com
<http://ovirt2-ks.test.j2noc.co=
m/>:/gluster-store/brick1/gv1=20
>>>>>>> Brick3: ovirt3-ks.test.j2noc.com
<http://ovirt3-ks.test.j2noc.co=
m/>:/gluster-store/brick1/gv1=20
>>>>>>>=20
>>>>>>> That's NIC 2's IP.=20
>>>>>>> Using 'iftop -i eno2 -L 5 -t' :=20
>>>>>>>=20
>>>>>>> dd if=3D/dev/zero of=3D/root/testfile bs=3D1M count=3D1000
oflag=
=3Ddirect=20
>>>>>>> 1048576000 bytes (1.0 GB) copied, 68.0714
s, 15.4 MB/s=20
>>>>>> Can you share the xml of this vm? You can find it in vdsm
log,=20
>>>>>> at the time you start the vm.=20
>>>>>>=20
>>>>>> Or you can do (on the host):=20
>>>>>>=20
>>>>>> # virsh=20
>>>>>> virsh # list=20
>>>>>> (username: vdsm@ovirt password: shibboleth)=20
>>>>>> virsh # dumpxml vm-id=20
>>>>>>=20
>>>>>>> Peak rate (sent/received/total): 281Mb
5.36Mb=
=20
>>>>>>> 282Mb=20
>>>>>>> Cumulative (sent/received/total): 1.96GB
14.6MB=
=20
>>>>>>> 1.97GB=20
>>>>>>>=20
>>>>>>> gluster volume info gv1:=20
>>>>>>> Options Reconfigured:=20
>>>>>>> performance.write-behind-window-size: 4MB=20
>>>>>>> performance.readdir-ahead: on=20
>>>>>>> performance.cache-size: 1GB=20
>>>>>>> performance.write-behind: off=20
>>>>>>>=20
>>>>>>> performance.write-behind: off didn't help.=20
>>>>>>> Neither did any other changes I've tried.=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> There is no VM traffic on this VM right now except my
test.=20
>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:=20
>>>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
<mailto:ravishank=
ar@redhat.com><ravishankar(a)redhat.com>
<mailto:ravishankar@redhat.com>=20
>>>>>>>> wrote:=20
>>>>>>>>> +gluster-users=20
>>>>>>>>>=20
>>>>>>>>> Does disabling 'performance.write-behind'
give a better throughp=
ut?=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:=20
>>>>>>>>>> I'm setting up a ovirt cluster using
glusterfs and noticing not=
stellar=20
>>>>>>>>>> performance.=20
>>>>>>>>>> Maybe my setup could use some adjustments?=20
>>>>>>>>>>=20
>>>>>>>>>> 3 hardware nodes running centos7.2, glusterfs
3.7.6.1, ovirt 3.=
6.2.6-1.=20
>>>>>>>>>> Each node has 8 spindles
configured in 1 array which is split u=
sing LVM=20
>>>>>>>>>> with one logical volume for
system and one for gluster.=20
>>>>>>>>>> They each have 4 NICs,=20
>>>>>>>>>> NIC1 =3D ovirtmgmt=20
>>>>>>>>>> NIC2 =3D gluster (1GbE)=20
>>>>>>>> How do you ensure that gluster trafic is using this
nic?=20
>>>>>>>>=20
>>>>>>>>>> NIC3 =3D VM traffic=20
>>>>>>>> How do you ensure that vm trafic is using this nic?=20
>>>>>>>>=20
>>>>>>>>>> I tried with default glusterfs settings=20
>>>>>>>> And did you find any difference?=20
>>>>>>>>=20
>>>>>>>>>> and also with:=20
>>>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>>>> performance.write-behind-window-size: 4MB=20
>>>>>>>>>>=20
>>>>>>>>>> [root@ovirt3 test scripts]# gluster volume info
gv1=20
>>>>>>>>>>=20
>>>>>>>>>> Volume Name: gv1=20
>>>>>>>>>> Type: Replicate=20
>>>>>>>>>> Volume ID:
71afc35b-09d7-4384-ab22-57d032a0f1a2=20
>>>>>>>>>> Status: Started=20
>>>>>>>>>> Number of Bricks: 1 x 3 =3D 3=20
>>>>>>>>>> Transport-type: tcp=20
>>>>>>>>>> Bricks:=20
>>>>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
<http://ovirt1-ks.test.j2noc.c=
om/>:/gluster-store/brick1/gv1=20
>>>>>>>>>> Brick2:
ovirt2-ks.test.j2noc.com <http://ovirt2-ks.test.j2noc.c=
om/>:/gluster-store/brick1/gv1=20
>>>>>>>>>> Brick3:
ovirt3-ks.test.j2noc.com <http://ovirt3-ks.test.j2noc.c=
om/>:/gluster-store/brick1/gv1=20
>>>>>>>>>> Options Reconfigured:=20
>>>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>>>> performance.write-behind-window-size: 4MB=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> Using simple dd test on VM in ovirt:=20
>>>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile bs=3D1G
count=3D1 ofl=
ag=3Ddirect=20
>>>>>>>> block size of 1G?!=20
>>>>>>>>=20
>>>>>>>> Try 1M (our default for storage operations)=20
>>>>>>>>=20
>>>>>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337
s, 16.3 MB/s=
=20
>>>>>>>>>>=20
>>>>>>>>>> Another VM not in ovirt using nfs:=20
>>>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile
bs=3D1G count=3D1 of=
lag=3Ddirect=20
>>>>>>>>>> 1073741824 bytes (1.1
GB) copied, 27.0079 s, 39.8 MB/s=
=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> Is that expected or is there a better way to set
it up to get b=
etter=20
>>>>>>>>>> performance?=20
>>>>>>>> Adding Niels for advice.=20
>>>>>>>>=20
>>>>>>>>>> This email, its contents and ....=20
>>>>>>>> Please avoid this, this is a public mailing list,
everything you =
write=20
>>>>>>>> here is public.=20
>>>>>>>>=20
>>>>>>>> Nir=20
>>>>>>> I'll have to look into how to remove this sig for this
mailing lis=
t....=20
>>>>>>>=20
>>>>>>> Cloud Services for Business
<http://www.j2.com/>www.j2.com <http:=
//www.j2.com/>=20
>>>>>>> j2 | eFax | eVoice | FuseMail |
Campaigner | KeepItSafe | Onebox=
=20
>>>>>>>=20
>>>>>>>=20
>>>>>>> This email, its contents and attachments contain information
from =
j2 Global,=20
>>>>>>> Inc. and/or its affiliates which may be
privileged, confidential o=
r=20
>>>>>>> otherwise protected from disclosure. The
information is intended t=
o be for=20
>>>>>>> the addressee(s) only. If you are not an
addressee, any disclosure=
, copy,=20
>>>>>>> distribution, or use of the contents of
this message is prohibited=
. If you=20
>>>>>>> have received this email in error please
notify the sender by repl=
y e-mail=20
>>>>>>> and delete the original message and any
copies. (c) 2015 j2 Global=
, Inc. All=20
>>>>>>> rights reserved. eFax, eVoice,
Campaigner, FuseMail, KeepItSafe, a=
nd Onebox=20
>>>>>>> are registered trademarks of j2 Global,
Inc. and its affiliates.=
=20
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>> http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.or=
g/mailman/listinfo/users>
l&utm_campaign=3Demployeeemail>
>>>> This email, its contents and attachments contain
information from j2 =
Global, Inc
<http://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-re=
ferral&utm_campaign=3Demployemail>. and/or its affiliates which may be priv=
ileged, confidential or otherwise protected from disclosure. The informatio=
n is intended to be for the addressee(s) only. If you are not an addressee,=
any disclosure, copy, distribution, or use of the contents of this message=
is prohibited. If you have received this email in error please notify the =
sender by reply e-mail and delete the original message and any copies. =A9 =
2015 j2 Global, Inc <http://www.j2.com/>;. All rights reserved. eFax =AE <ht=
tp://www.efax.com/>;, eVoice =AE <http://www.evoice.com/>;, Campaigner =AE <h=
ttp://www.campaigner.com/>;, FuseMail =AE <http://www.fusemail.com/>;, KeepIt=
Safe =AE <http://www.keepitsafe.com/> and Onebox =AE <http://www.onebox.com=
/> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. and i=
ts affiliates.
>>>>=20
>>>>=20
>>>>=20
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org=
/mailman/listinfo/users>
&utm_campaign=3Demployeeemail>
>>> This email, its contents and attachments contain
information from j2 G=
lobal, Inc
<http://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-ref=
erral&utm_campaign=3Demployemail>. and/or its affiliates which may be privi=
leged, confidential or otherwise protected from disclosure. The information=
is intended to be for the addressee(s) only. If you are not an addressee, =
any disclosure, copy, distribution, or use of the contents of this message =
is prohibited. If you have received this email in error please notify the s=
ender by reply e-mail and delete the original message and any copies. =A9 2=
015 j2 Global, Inc <http://www.j2.com/>;. All rights reserved. eFax =AE <htt=
p://www.efax.com/>;, eVoice =AE <http://www.evoice.com/>;, Campaigner =AE <ht=
tp://www.campaigner.com/>;, FuseMail =AE <http://www.fusemail.com/>;, KeepItS=
afe =AE <http://www.keepitsafe.com/> and Onebox =AE <http://www.onebox.com/=
are r egistered trademarks of j2 Global, Inc
<http://www.j2.com/>;. and it=
s affiliates.
>>>=20
>>>=20
>>>=20
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/=
mailman/listinfo/users>
>>=20
>>=20
>>=20
>> --=20
>> This message is subject to the CSIR's copyright terms and conditions, e=
-mail legal notice, and implemented Open Document Format (ODF) standard.=20
>> The full disclaimer details can be found at
<http://www.csir.co.za/dis=
claimer.html>http://www.csir.co.za/disclaimer.html
<http://www.csir.co.za/d=
isclaimer.html>.
>>=20
>> This message has been scanned for viruses and dangerous content by Mail=
Scanner <http://www.mailscanner.info/>,=20
>> and is believed to be clean.
>>=20
>>=20
>> Please consider the environment before printing this email.
>>=20
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/m=
ailman/listinfo/users>
>=20
>=20
> --=20
> This message is subject to the CSIR's copyright terms and conditions, e-=
mail legal notice, and implemented Open Document Format (ODF) standard.=20
> The full disclaimer details can be found at
<http://www.csir.co.za/disc=
laimer.html>http://www.csir.co.za/disclaimer.html
<http://www.csir.co.za/di=
sclaimer.html>.
>=20
> This message has been scanned for viruses and dangerous content by MailS=
canner <http://www.mailscanner.info/>,=20
> and is believed to be clean.
>=20
>=20
> Please consider the environment before printing this email.
>=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--=20
This message is subject to the CSIR's copyright terms and conditions, e-mai=
l legal notice, and implemented Open Document Format (ODF) standard.=20
The full disclaimer details can be found at http://www.csir.co.za/disclaime=
r.html.
This message has been scanned for viruses and dangerous content by MailScan=
ner,=20
and is believed to be clean.
Please consider the environment before printing this email.
--Apple-Mail=_2474E8FB-D4B8-41DB-B4A1-843303C4F112
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html charset=
=3Dwindows-1252"></head><body style=3D"word-wrap: break-word;
-webkit-nbsp-=
mode: space; -webkit-line-break: after-white-space;"
class=3D"">Hi<div clas=
s=3D""><br class=3D""></div><div
class=3D""><blockquote type=3D"cite" class=
=3D""><div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">It is not remove=
d. Can you try 'gluster volume set volname cluster.eager-lock enable`?</div=
</blockquote><div class=3D""><br
class=3D""></div>This works. BTW by defau=
lt this setting is
=93on=94. What=92s the difference between =93on=94 and =
=93enable=94?</div><div class=3D""><br
class=3D""></div><div class=3D"">Tha=
nks for the clarification.</div><div class=3D""><br
class=3D""></div><div c=
lass=3D"">Regards,</div><div class=3D""><br
class=3D""></div><div class=3D"=
">Roderick</div><div class=3D""><br
class=3D""><div><blockquote type=3D"cit=
e" class=3D""><div class=3D"">On 06 Apr 2016, at 10:56
AM, Ravishankar N &l=
t;<a href=3D"mailto:ravishankar@redhat.com"
class=3D"">ravishankar(a)redhat.c=
om</a>> wrote:</div><br
class=3D"Apple-interchange-newline"><div class=
=3D"">
=20=20
<meta content=3D"text/html; charset=3Dwindows-1252"
http-equiv=3D"Conte=
nt-Type" class=3D"">
=20=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">
<div class=3D"moz-cite-prefix">On 04/06/2016 02:08 PM, Roderick Mooi
wrote:<br class=3D"">
</div>
<blockquote cite=3D"mid:90A2411F-0F73-437D-899B-6098A5C7A1B6@csir.co.za=
" type=3D"cite" class=3D"">
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3Dwindows-1252" class=3D"">
Hi Ravi and colleagues
<div class=3D""><br class=3D"">
</div>
<div class=3D"">(apologies for hijacking this thread but I=92m not
sure where else to report this (and it is related).)
<div class=3D""><br class=3D"">
</div>
<div class=3D"">With gluster 3.7.10, running</div>
<div class=3D"">#gluster volume set <volname> group
virt</div>
<div class=3D"">fails with:</div>
<div class=3D"">volume set: failed: option : eager-lock does not
exist</div>
<div class=3D"">Did you mean eager-lock?</div>
<div class=3D""><br class=3D"">
</div>
<div class=3D"">I had to remove the eager-lock setting from
/var/lib/glusterd/groups/virt to get this to work. It seems
like setting eager-lock has been removed from latest gluster.
Is this correct? Either way, is there anything else I should
do?</div>
</div>
</blockquote>
<br class=3D"">
It is not removed. Can you try 'gluster volume set volname
cluster.eager-lock enable`?<br class=3D"">
I think the disperse (EC) translator introduced a
`disperse.eager-lock` which is why you would need to mention entire
volume option name to avoid ambiguity.<br class=3D"">
We probably need to fix the virt profile setting to include the
entire name. By the way 'gluster volume set help` should give you
the list of all options.<br class=3D"">
<br class=3D"">
-Ravi<br class=3D"">
<br class=3D"">
<blockquote cite=3D"mid:90A2411F-0F73-437D-899B-6098A5C7A1B6@csir.co.za=
" type=3D"cite" class=3D"">
<div class=3D"">
<div class=3D""><br class=3D"">
</div>
<div class=3D"">Cheers,</div>
<div class=3D""><br class=3D"">
</div>
<div class=3D"">Roderick<br class=3D"">
<div class=3D""><br class=3D"">
<div class=3D"">
<blockquote type=3D"cite" class=3D"">
<div class=3D"">On 12 Feb 2016, at 6:18 AM, Ravishankar N
<<a moz-do-not-send=3D"true"
href=3D"mailto:ravishanka=
r(a)redhat.com" class=3D"">ravishankar(a)redhat.com</a>&gt;
wrote:</div>
<br class=3D"Apple-interchange-newline">
<div class=3D"">
<meta content=3D"text/html; charset=3Dwindows-1252" http-=
equiv=3D"Content-Type" class=3D"">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">
<div class=3D"moz-cite-prefix">Hi Bill,<br
class=3D"">
Can you enable virt-profile setting for your
volume and see if that helps? You need to enable
this optimization when you create the volume using
ovrit, or use the following command for an
existing volume:<br class=3D"">
<tt class=3D""><br class=3D"">
#gluster volume set <volname> group
virt</tt>=
<br class=3D"">
<br class=3D"">
-Ravi<br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/12/2016 05:22 AM, Bill James wrote:<br class=3D=
"">
</div>
<blockquote cite=3D"mid:56BD1EB9.5000802@j2.com" type=
=3D"cite" class=3D"">
<meta content=3D"text/html; charset=3Dwindows-1252" h=
ttp-equiv=3D"Content-Type" class=3D"">
My apologies, I'm showing how much of a noob I am.<br=
class=3D"">
Ignore last direct to gluster numbers, as that
wasn't really glusterfs.<br class=3D"">
<br class=3D"">
<br class=3D"">
[root@ovirt2 test ~]# mount -t glusterfs <a moz-do-no=
t-send=3D"true" href=3D"http://ovirt2-ks.test.j2noc.com/"
class=3D"">ovirt2=
-ks.test.j2noc.com</a>:/gv1
/mnt/tmp/<br class=3D"">
[root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/mnt/tmp/testfile2 bs=3D1M count=3D1000
oflag=3Ddirect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9
MB/s<br class=3D"">
<br class=3D"">
That's more how I expected, it is pointing to
glusterfs performance.<br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 02/11/2016 03:27
PM, Bill James wrote:<br class=3D"">
</div>
<blockquote cite=3D"mid:56BD18D7.8080909@j2.com" type=
=3D"cite" class=3D"">
<meta http-equiv=3D"Content-Type"
content=3D"text/h=
tml; charset=3Dwindows-1252" class=3D"">
don't know if it helps, but I ran a few more
tests, all from the same hardware node.<br class=3D=
"">
<br class=3D"">
The VM:<br class=3D"">
[root@billjov1 ~]# time dd if=3D/dev/zero
of=3D/root/testfile bs=3D1M count=3D1000 oflag=3Ddi=
rect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 62.5535 s,
16.8 MB/s<br class=3D"">
<br class=3D"">
Writing directly to gluster volume:<br
class=3D"">
[root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/gluster-store/brick1/gv1/testfile bs=3D1M
count=3D1000 oflag=3Ddirect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 9.92048 s, 106
MB/s<br class=3D"">
<br class=3D"">
<br class=3D"">
Writing to NFS volume:<br class=3D"">
[root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/mnt/storage/qa/testfile bs=3D1M count=3D1000
oflag=3Ddirect<br class=3D"">
1048576000 bytes (1.0 GB) copied, 10.5776 s,
99.1 MB/s<br class=3D"">
<br class=3D"">
NFS & Gluster are using the same interface.
Tests were not run at same time.<br class=3D"">
<br class=3D"">
This would suggest my problem isn't glusterfs,
but the VM performance.<br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 02/11/2016
03:13
PM, Bill James wrote:<br class=3D"">
</div>
<blockquote cite=3D"mid:56BD1589.8090506@j2.com" ty=
pe=3D"cite" class=3D"">xml attached. <br
class=3D"">
<br class=3D"">
<br class=3D"">
On 02/11/2016 12:28 PM, Nir Soffer wrote: <br cla=
ss=3D"">
<blockquote type=3D"cite"
class=3D"">On Thu, Feb
11, 2016 at 8:27 PM, Bill James <a moz-do-not-s=
end=3D"true" class=3D"moz-txt-link-rfc2396E"
href=3D"mailto:bill.james@j2.c=
om"></a><a class=3D"moz-txt-link-rfc2396E"
href=3D"mailto:bill.james@j2.com=
">&lt;bill.james(a)j2.com&gt;</a>
wrote: <br class=3D"">
<blockquote type=3D"cite"
class=3D"">thank you
for the reply. <br class=3D"">
<br class=3D"">
We setup gluster using the names
associated with NIC 2 IP. <br class=3D"=
">
Brick1: <a moz-do-not-send=3D"true"
hr=
ef=3D"http://ovirt1-ks.test.j2noc.com/"
class=3D"">ovirt1-ks.test.j2noc.com=
</a>:/gluster-store/brick1/gv1
<br class=3D"">
Brick2: <a moz-do-not-send=3D"true"
hr=
ef=3D"http://ovirt2-ks.test.j2noc.com/"
class=3D"">ovirt2-ks.test.j2noc.com=
</a>:/gluster-store/brick1/gv1
<br class=3D"">
Brick3: <a moz-do-not-send=3D"true"
hr=
ef=3D"http://ovirt3-ks.test.j2noc.com/"
class=3D"">ovirt3-ks.test.j2noc.com=
</a>:/gluster-store/brick1/gv1
<br class=3D"">
<br class=3D"">
That's NIC 2's IP. <br class=3D"">
Using 'iftop -i eno2 -L 5 -t' : <br
class=3D"=
">
<br class=3D"">
dd if=3D/dev/zero of=3D/root/testfile bs=3D1M
count=3D1000 oflag=3Ddirect <br class=3D"">
1048576000 bytes (1.0 GB) copied, 68.0714
s, 15.4 MB/s <br class=3D"">
</blockquote>
Can you share the xml of this vm? You can
find it in vdsm log, <br class=3D"">
at the time you start the vm. <br class=3D"">
<br class=3D"">
Or you can do (on the host): <br class=3D"">
<br class=3D"">
# virsh <br class=3D"">
virsh # list <br class=3D"">
(username: vdsm@ovirt password: shibboleth)
<br class=3D"">
virsh # dumpxml vm-id <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">Peak rate
(sent/received/total):  =
; &n=
bsp;
281Mb 5.36Mb <br class=3D"">
282Mb <br class=3D"">
Cumulative
(sent/received/total):  =
; &n=
bsp;
1.96GB 14.6MB <br class=3D"">
1.97GB <br class=3D"">
<br class=3D"">
gluster volume info gv1: <br class=3D"">
Options Reconfigured: <br
class=3D"">
performance.write-behind-window-size:
4MB <br class=3D"">
performance.readdir-ahead: on <br clas=
s=3D"">
performance.cache-size: 1GB <br class=
=3D"">
performance.write-behind: off <br clas=
s=3D"">
<br class=3D"">
performance.write-behind: off didn't help.
<br class=3D"">
Neither did any other changes I've tried.
<br class=3D"">
<br class=3D"">
<br class=3D"">
There is no VM traffic on this VM right
now except my test. <br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/10/2016 11:55 PM, Nir Soffer wrote:
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">On Thu,
Feb 11, 2016 at 2:42 AM, Ravishankar N <a m=
oz-do-not-send=3D"true" class=3D"moz-txt-link-rfc2396E"
href=3D"mailto:ravi=
shankar(a)redhat.com"></a><a class=3D"moz-txt-link-rfc2396E"
href=3D"mailto:r=
avishankar@redhat.com">&lt;ravishankar(a)redhat.com&gt;</a>
<br class=3D"">
wrote: <br class=3D"">
<blockquote type=3D"cite"
class=3D"">+glust=
er-users
<br class=3D"">
<br class=3D"">
Does disabling
'performance.write-behind' give a
better throughput? <br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
On 02/10/2016 11:06 PM, Bill James
wrote: <br class=3D"">
<blockquote type=3D"cite"
class=3D"">I'm
setting up a ovirt cluster using
glusterfs and noticing not stellar <br =
class=3D"">
performance. <br class=3D"">
Maybe my setup could use some
adjustments? <br class=3D"">
<br class=3D"">
3 hardware nodes running centos7.2,
glusterfs 3.7.6.1, ovirt 3.6.2.6-1.
<br class=3D"">
Each node has 8 spindles configured
in 1 array which is split using LVM
<br class=3D"">
with one logical volume for system
and one for gluster. <br class=3D"">
They each have 4 NICs, <br
class=3D"">
NIC1 =3D ovirtmgmt <br cla=
ss=3D"">
NIC2 =3D gluster (1G=
bE) <br class=3D"">
</blockquote>
</blockquote>
How do you ensure that gluster trafic is
using this nic? <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">
<blockquote type=3D"cite"
class=3D"">&nbs=
p;
NIC3 =3D VM traffic <br class=3D"">
</blockquote>
</blockquote>
How do you ensure that vm trafic is
using this nic? <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">
<blockquote type=3D"cite"
class=3D"">I
tried with default glusterfs
settings <br class=3D"">
</blockquote>
</blockquote>
And did you find any difference? <br class=
=3D"">
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">
<blockquote type=3D"cite"
class=3D"">and
also with: <br class=3D"">
performance.cache-size: 1GB <br class=
=3D"">
performance.readdir-ahead: on <br class=
=3D"">
performance.write-behind-window-size:
4MB <br class=3D"">
<br class=3D"">
[root@ovirt3 test scripts]# gluster
volume info gv1 <br class=3D"">
<br class=3D"">
Volume Name: gv1 <br class=3D"">
Type: Replicate <br class=3D"">
Volume ID:
71afc35b-09d7-4384-ab22-57d032a0f1a2
<br class=3D"">
Status: Started <br class=3D"">
Number of Bricks: 1 x 3 =3D 3 <br class=
=3D"">
Transport-type: tcp <br class=3D"">
Bricks: <br class=3D"">
Brick1: <a moz-do-not-send=3D"true" hre=
f=3D"http://ovirt1-ks.test.j2noc.com/"
class=3D"">ovirt1-ks.test.j2noc.com<=
/a>:/gluster-store/brick1/gv1
<br class=3D"">
Brick2: <a moz-do-not-send=3D"true" hre=
f=3D"http://ovirt2-ks.test.j2noc.com/"
class=3D"">ovirt2-ks.test.j2noc.com<=
/a>:/gluster-store/brick1/gv1
<br class=3D"">
Brick3: <a moz-do-not-send=3D"true" hre=
f=3D"http://ovirt3-ks.test.j2noc.com/"
class=3D"">ovirt3-ks.test.j2noc.com<=
/a>:/gluster-store/brick1/gv1
<br class=3D"">
Options Reconfigured: <br class=3D"">
performance.cache-size: 1GB <br class=
=3D"">
performance.readdir-ahead: on <br class=
=3D"">
performance.write-behind-window-size:
4MB <br class=3D"">
<br class=3D"">
<br class=3D"">
Using simple dd test on VM in ovirt:
<br class=3D"">
dd if=3D/dev/zero
of=3D/root/testfile bs=3D1G count=3D1
oflag=3Ddirect <br class=3D"">
</blockquote>
</blockquote>
block size of 1G?! <br class=3D"">
<br class=3D"">
Try 1M (our default for storage
operations) <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">
<blockquote type=3D"cite"
class=3D"">&nbs=
p;
1073741824 bytes (1.1 GB) copied,
65.9337 s, 16.3 MB/s <br class=3D"">
<br class=3D"">
Another VM not in ovirt using nfs: <br =
class=3D"">
dd
if=3D/dev/z=
ero
of=3D/root/testfile bs=3D1G count=3D1
oflag=3Ddirect <br class=3D"">
&nb=
sp; 1073741824 bytes (1.1 GB)
copied, 27.0079 s, 39.8 MB/s <br class=
=3D"">
<br class=3D"">
<br class=3D"">
Is that expected or is there a
better way to set it up to get
better <br class=3D"">
performance? <br class=3D"">
</blockquote>
</blockquote>
Adding Niels for advice. <br class=3D"">
<br class=3D"">
<blockquote type=3D"cite"
class=3D"">
<blockquote type=3D"cite"
class=3D"">This
email, its contents and .... <br class=
=3D"">
</blockquote>
</blockquote>
Please avoid this, this is a public
mailing list, everything you write <br clas=
s=3D"">
here is public. <br class=3D"">
<br class=3D"">
Nir <br class=3D"">
</blockquote>
I'll have to look into how to remove this
sig for this mailing list.... <br
class=3D"">
<br class=3D"">
Cloud Services for Business <a moz-do-not-sen=
d=3D"true" class=3D"moz-txt-link-abbreviated"
href=3D"http://www.j2.com/"><=
/a><a class=3D"moz-txt-link-abbreviated"
href=3D"http://www.j2.com/">www.j2=
.com</a>
<br class=3D"">
j2 | eFax | eVoice | FuseMail | Campaigner
| KeepItSafe | Onebox <br class=3D"">
<br class=3D"">
<br class=3D"">
This email, its contents and attachments
contain information from j2 Global, <br class=
=3D"">
Inc. and/or its affiliates which may be
privileged, confidential or <br class=3D"">
otherwise protected from disclosure. The
information is intended to be for <br class=
=3D"">
the addressee(s) only. If you are not an
addressee, any disclosure, copy, <br class=3D=
"">
distribution, or use of the contents of
this message is prohibited. If you <br class=
=3D"">
have received this email in error please
notify the sender by reply e-mail <br class=
=3D"">
and delete the original message and any
copies. (c) 2015 j2 Global, Inc. All <br clas=
s=3D"">
rights reserved. eFax, eVoice, Campaigner,
FuseMail, KeepItSafe, and Onebox <br class=3D=
"">
are registered trademarks of j2 Global,
Inc. and its affiliates. <br class=3D"">
</blockquote>
</blockquote>
<br class=3D"">
<br class=3D"">
<fieldset
class=3D"mimeAttachmentHeader"></fields=
et>
<br class=3D"">
<pre class=3D""
wrap=3D"">_______________________=
________________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated"
href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext"
href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ma...
info/users</a>
</pre>
</blockquote>
<br class=3D""><p class=3D""><a
moz-do-not-send=3D"=
true"
href=3D"http://www.j2.com/?utm_source=3Dj2global&utm_medium=...
ll-referral&utm_campaign=3Demployeeemail" class=3D""><span
style=3D"col=
or:windowtext;
text-decoration:none" class=3D""><img
moz-do-=
not-send=3D"true"
src=3D"http://home.j2.com/j2_Global_Cloud_Services/j2_Glo=
bal_Email_Footer.jpg" alt=3D"www.j2.com" class=3D""
height=3D"46" width=3D"=
391" border=3D"0"></span></a></p><p
class=3D""><span style=3D"font-size:8.0=
pt;font-family:"Arial","sans-serif";color:gray"
class=
=3D"">This email, its contents and
attachments contain information from <a moz-do-=
not-send=3D"true"
href=3D"http://www.j2.com/?utm_source=3Dj2global&utm_=
medium=3Dxsell-referral&utm_campaign=3Demployemail"
class=3D"">j2 Globa=
l, Inc</a>. and/or its
affiliates which may be privileged,
confidential or otherwise protected from
disclosure. The information is intended to
be for the addressee(s) only. If you are not
an addressee, any disclosure, copy,
distribution, or use of the contents of this
message is prohibited. If you have received
this email in error please notify the sender
by reply e-mail and delete the original
message and any copies. =A9 2015 <a moz-do-not-=
send=3D"true" href=3D"http://www.j2.com/" class=3D"">j2
Global, Inc</a>. All rights reserved. <a moz-=
do-not-send=3D"true" href=3D"http://www.efax.com/"
class=3D"">eFax
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"=
http://www.evoice.com/" class=3D"">eVoice
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"=
http://www.campaigner.com/" class=3D"">Campaigner
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"=
http://www.fusemail.com/" class=3D"">FuseMail
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"=
http://www.keepitsafe.com/" class=3D"">KeepItSafe
=AE</a> and <a moz-do-not-send=3D"true"
href=
=3D"http://www.onebox.com/" class=3D"">Onebox
=AE</a> are r egistered trademarks of <a moz-=
do-not-send=3D"true" href=3D"http://www.j2.com/"
class=3D"">j2
Global, Inc</a>. and its
affiliates.</span></=
p>
<br class=3D"">
<fieldset
class=3D"mimeAttachmentHeader"></fieldset>
<br class=3D"">
<pre class=3D""
wrap=3D"">_________________________=
______________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated"
href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext"
href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ma...
info/users</a>
</pre>
</blockquote>
<br class=3D""><p class=3D""><a
moz-do-not-send=3D"tr=
ue"
href=3D"http://www.j2.com/?utm_source=3Dj2global&utm_medium=...
-referral&utm_campaign=3Demployeeemail" class=3D""><span
style=3D"color=
:windowtext;
text-decoration:none" class=3D""><img
moz-do-no=
t-send=3D"true"
src=3D"http://home.j2.com/j2_Global_Cloud_Services/j2_Globa=
l_Email_Footer.jpg" alt=3D"www.j2.com" class=3D""
height=3D"46" width=3D"39=
1" border=3D"0"></span></a></p><p
class=3D""><span style=3D"font-size:8.0pt=
;font-family:"Arial","sans-serif";color:gray"
class=3D"=
">This email, its contents and
attachments contain information from <a moz-do-no=
t-send=3D"true"
href=3D"http://www.j2.com/?utm_source=3Dj2global&utm_me=
dium=3Dxsell-referral&utm_campaign=3Demployemail" class=3D"">j2
Global,=
Inc</a>. and/or its
affiliates which may be privileged,
confidential or otherwise protected from
disclosure. The information is intended to be
for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution,
or use of the contents of this message is
prohibited. If you have received this email in
error please notify the sender by reply e-mail
and delete the original message and any
copies. =A9 2015 <a moz-do-not-send=3D"true" href=
=3D"http://www.j2.com/" class=3D"">j2
Global, Inc</a>. All rights reserved. <a moz-do=
-not-send=3D"true" href=3D"http://www.efax.com/"
class=3D"">eFax =AE</a>,
<a moz-do-not-send=3D"true"
href=3D"http://www.ev=
oice.com/" class=3D"">eVoice
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"ht=
tp://www.campaigner.com/" class=3D"">Campaigner
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"ht=
tp://www.fusemail.com/" class=3D"">FuseMail
=AE</a>, <a moz-do-not-send=3D"true"
href=3D"ht=
tp://www.keepitsafe.com/" class=3D"">KeepItSafe
=AE</a> and <a moz-do-not-send=3D"true"
href=3D=
"http://www.onebox.com/" class=3D"">Onebox
=AE</a> are r egistered trademarks of <a moz-do=
-not-send=3D"true" href=3D"http://www.j2.com/"
class=3D"">j2
Global, Inc</a>. and its
affiliates.</span></p>
<br class=3D"">
<fieldset
class=3D"mimeAttachmentHeader"></fieldset>
<br class=3D"">
<pre class=3D""
wrap=3D"">___________________________=
____________________
Users mailing list
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-abbreviated"
href=3D"mail=
to:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send=3D"true" class=3D"moz-txt-link-freetext"
href=3D"http://=
lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/ma...
info/users</a>
</pre>
</blockquote>
<br class=3D"">
<br class=3D"">
<font class=3D""
face=3D"Verdana,Arial,Helvetica,Trebuc=
het MS" size=3D"1">
<br class=3D"">
-- <br class=3D"">
This message is subject to the CSIR's copyright
terms and conditions, e-mail legal notice, and
implemented Open Document Format (ODF) standard.
<br class=3D"">
The full disclaimer details can be found at <a moz-do=
-not-send=3D"true" href=3D"http://www.csir.co.za/disclaimer.html"
class=3D"=
"></a><a class=3D"moz-txt-link-freetext"
href=3D"http://www.csir.co.za/disc=
laimer.html">http://www.csir.co.za/disclaimer.html</a>.
<p class=3D"">
<br class=3D"">
This message has been scanned for viruses and
dangerous content by <a moz-do-not-send=3D"true" hr=
ef=3D"http://www.mailscanner.info/" class=3D""><b
class=3D"">MailScanner</b=
</a>, <br class=3D"">
and is believed to be clean.
</p>
</font><p class=3D""><font
class=3D"" face=3D"Verdana,A=
rial,Helvetica,Trebuchet MS" size=3D"1">
<br class=3D"">
Please consider the environment before printing
this email.
</font>
</p>
</div>
_______________________________________________<br class=
=3D"">
Users mailing list<br class=3D"">
<a moz-do-not-send=3D"true"
href=3D"mailto:Users@ovirt.or=
g" class=3D"">Users(a)ovirt.org</a><br class=3D"">
<a class=3D"moz-txt-link-freetext"
href=3D"http://lists.o=
virt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/l...
ers</a><br class=3D"">
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</div>
</div>
<font face=3D"Verdana,Arial,Helvetica,Trebuchet MS"
size=3D"1" class=
=3D"">
<br class=3D"">
-- <br class=3D"">
This message is subject to the CSIR's copyright terms and
conditions, e-mail legal notice, and implemented Open Document
Format (ODF) standard.
<br class=3D"">
The full disclaimer details can be found at <a moz-do-not-send=3D"t=
rue" href=3D"http://www.csir.co.za/disclaimer.html"
class=3D""></a><a class=
=3D"moz-txt-link-freetext"
href=3D"http://www.csir.co.za/disclaimer.html">h=
ttp://www.csir.co.za/disclaimer.html</a>.
<p class=3D"">
<br class=3D"">
This message has been scanned for viruses and dangerous
content by <a moz-do-not-send=3D"true"
href=3D"http://www.mailsca=
nner.info/" class=3D""><b
class=3D"">MailScanner</b></a>,
<br class=3D"">
and is believed to be clean.
</p>
</font><p class=3D""><font
face=3D"Verdana,Arial,Helvetica,Trebuchet =
MS" size=3D"1" class=3D"">
<br class=3D"">
Please consider the environment before printing this email.
</font>
</p>
</blockquote>
<br class=3D"">
<br class=3D"">
</div>
_______________________________________________<br class=3D"">Users
mailing=
list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
class=3D"">Users@ovi=
rt.org</a><br
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br c=
lass=3D""></div></blockquote></div><br
class=3D""></div><font face=3D"Verda=
na,Arial,Helvetica,Trebuchet MS" size=3D"1">
<br />--=20
<br />This message is subject to the CSIR's copyright terms and conditions,=
e-mail legal notice, and implemented Open Document Format (ODF) standard.
<br />The full disclaimer details can be found at <a
href=3D"http://www.csi=
r.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a>.
<p>
<br />This message has been scanned for viruses and dangerous content by <a=
href=3D"http://www.mailscanner.info/"><b>MailScanner</b></a>,=20
<br />and is believed to be clean.
<p>
<br />Please consider the environment before printing this email.
</font>
</body></html>
--Apple-Mail=_2474E8FB-D4B8-41DB-B4A1-843303C4F112--
12:37 p.m.
This is a multi-part message in MIME format.
--------------020300020609000301040107
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
On 04/12/2016 02:41 PM, Roderick Mooi wrote:
Hi
> It is not removed. Can you try 'gluster volume set volname
> cluster.eager-lock enable`?
This works. BTW by default this setting is on. Whats the difference
between on and enable?
Both are identical. You can use any of the booleans to achieve the same
effect. {"1", "on", "yes", "true",
"enable"} or {"0", "off", "no",
"false", "disable"}
FYI, the patch http://review.gluster.org/#/c/13958/ to fix this issue
should make it to glusterfs-3.7.11.
-Ravi
Thanks for the clarification.
Regards,
Roderick
> On 06 Apr 2016, at 10:56 AM, Ravishankar N <ravishankar(a)redhat.com
> <mailto:ravishankar@redhat.com>> wrote:
>
> On 04/06/2016 02:08 PM, Roderick Mooi wrote:
>> Hi Ravi and colleagues
>>
>> (apologies for hijacking this thread but Im not sure where else to
>> report this (and it is related).)
>>
>> With gluster 3.7.10, running
>> #gluster volume set <volname> group virt
>> fails with:
>> volume set: failed: option : eager-lock does not exist
>> Did you mean eager-lock?
>>
>> I had to remove the eager-lock setting from
>> /var/lib/glusterd/groups/virt to get this to work. It seems like
>> setting eager-lock has been removed from latest gluster. Is this
>> correct? Either way, is there anything else I should do?
>
> It is not removed. Can you try 'gluster volume set volname
> cluster.eager-lock enable`?
> I think the disperse (EC) translator introduced a
> `disperse.eager-lock` which is why you would need to mention entire
> volume option name to avoid ambiguity.
> We probably need to fix the virt profile setting to include the
> entire name. By the way 'gluster volume set help` should give you the
> list of all options.
>
> -Ravi
>
>>
>> Cheers,
>>
>> Roderick
>>
>>> On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishankar(a)redhat.com
>>> <mailto:ravishankar@redhat.com>> wrote:
>>>
>>> Hi Bill,
>>> Can you enable virt-profile setting for your volume and see if that
>>> helps? You need to enable this optimization when you create the
>>> volume using ovrit, or use the following command for an existing
>>> volume:
>>>
>>> #gluster volume set <volname> group virt
>>>
>>> -Ravi
>>>
>>>
>>> On 02/12/2016 05:22 AM, Bill James wrote:
>>>> My apologies, I'm showing how much of a noob I am.
>>>> Ignore last direct to gluster numbers, as that wasn't really
>>>> glusterfs.
>>>>
>>>>
>>>> [root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com
>>>> <http://ovirt2-ks.test.j2noc.com/>:/gv1 /mnt/tmp/
>>>> [root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
>>>> bs=1M count=1000 oflag=direct
>>>> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
>>>>
>>>> That's more how I expected, it is pointing to glusterfs performance.
>>>>
>>>>
>>>>
>>>> On 02/11/2016 03:27 PM, Bill James wrote:
>>>>> don't know if it helps, but I ran a few more tests, all from the
>>>>> same hardware node.
>>>>>
>>>>> The VM:
>>>>> [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M
>>>>> count=1000 oflag=direct
>>>>> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
>>>>>
>>>>> Writing directly to gluster volume:
>>>>> [root@ovirt2 test ~]# time dd if=/dev/zero
>>>>> of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
>>>>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
>>>>>
>>>>>
>>>>> Writing to NFS volume:
>>>>> [root@ovirt2 test ~]# time dd if=/dev/zero
>>>>> of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct
>>>>> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
>>>>>
>>>>> NFS & Gluster are using the same interface. Tests were not run at
>>>>> same time.
>>>>>
>>>>> This would suggest my problem isn't glusterfs, but the VM
>>>>> performance.
>>>>>
>>>>>
>>>>>
>>>>> On 02/11/2016 03:13 PM, Bill James wrote:
>>>>>> xml attached.
>>>>>>
>>>>>>
>>>>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:
>>>>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James
<bill.james(a)j2.com>
>>>>>>> wrote:
>>>>>>>> thank you for the reply.
>>>>>>>>
>>>>>>>> We setup gluster using the names associated with NIC 2
IP.
>>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
>>>>>>>>
<http://ovirt1-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
>>>>>>>> Brick2: ovirt2-ks.test.j2noc.com
>>>>>>>>
<http://ovirt2-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
>>>>>>>> Brick3: ovirt3-ks.test.j2noc.com
>>>>>>>>
<http://ovirt3-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
>>>>>>>>
>>>>>>>> That's NIC 2's IP.
>>>>>>>> Using 'iftop -i eno2 -L 5 -t' :
>>>>>>>>
>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1M count=1000
oflag=direct
>>>>>>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
>>>>>>> Can you share the xml of this vm? You can find it in vdsm
log,
>>>>>>> at the time you start the vm.
>>>>>>>
>>>>>>> Or you can do (on the host):
>>>>>>>
>>>>>>> # virsh
>>>>>>> virsh # list
>>>>>>> (username: vdsm@ovirt password: shibboleth)
>>>>>>> virsh # dumpxml vm-id
>>>>>>>
>>>>>>>> Peak rate (sent/received/total): 281Mb 5.36Mb
>>>>>>>> 282Mb
>>>>>>>> Cumulative (sent/received/total): 1.96GB 14.6MB
>>>>>>>> 1.97GB
>>>>>>>>
>>>>>>>> gluster volume info gv1:
>>>>>>>> Options Reconfigured:
>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>> performance.readdir-ahead: on
>>>>>>>> performance.cache-size: 1GB
>>>>>>>> performance.write-behind: off
>>>>>>>>
>>>>>>>> performance.write-behind: off didn't help.
>>>>>>>> Neither did any other changes I've tried.
>>>>>>>>
>>>>>>>>
>>>>>>>> There is no VM traffic on this VM right now except my
test.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>>>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
>>>>>>>>> <ravishankar(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>> +gluster-users
>>>>>>>>>>
>>>>>>>>>> Does disabling 'performance.write-behind'
give a better
>>>>>>>>>> throughput?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>>>>>>>> I'm setting up a ovirt cluster using
glusterfs and noticing
>>>>>>>>>>> not stellar
>>>>>>>>>>> performance.
>>>>>>>>>>> Maybe my setup could use some adjustments?
>>>>>>>>>>>
>>>>>>>>>>> 3 hardware nodes running centos7.2, glusterfs
3.7.6.1,
>>>>>>>>>>> ovirt 3.6.2.6-1.
>>>>>>>>>>> Each node has 8 spindles configured in 1
array which is
>>>>>>>>>>> split using LVM
>>>>>>>>>>> with one logical volume for system and one
for gluster.
>>>>>>>>>>> They each have 4 NICs,
>>>>>>>>>>> NIC1 = ovirtmgmt
>>>>>>>>>>> NIC2 = gluster (1GbE)
>>>>>>>>> How do you ensure that gluster trafic is using this
nic?
>>>>>>>>>
>>>>>>>>>>> NIC3 = VM traffic
>>>>>>>>> How do you ensure that vm trafic is using this nic?
>>>>>>>>>
>>>>>>>>>>> I tried with default glusterfs settings
>>>>>>>>> And did you find any difference?
>>>>>>>>>
>>>>>>>>>>> and also with:
>>>>>>>>>>> performance.cache-size: 1GB
>>>>>>>>>>> performance.readdir-ahead: on
>>>>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>>>>
>>>>>>>>>>> [root@ovirt3 test scripts]# gluster volume
info gv1
>>>>>>>>>>>
>>>>>>>>>>> Volume Name: gv1
>>>>>>>>>>> Type: Replicate
>>>>>>>>>>> Volume ID:
71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>>>>>>>> Status: Started
>>>>>>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>>>>>>> Transport-type: tcp
>>>>>>>>>>> Bricks:
>>>>>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
>>>>>>>>>>>
<http://ovirt1-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
>>>>>>>>>>> Brick2: ovirt2-ks.test.j2noc.com
>>>>>>>>>>>
<http://ovirt2-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
>>>>>>>>>>> Brick3: ovirt3-ks.test.j2noc.com
>>>>>>>>>>>
<http://ovirt3-ks.test.j2noc.com/>:/gluster-store/brick1/gv1
>>>>>>>>>>> Options Reconfigured:
>>>>>>>>>>> performance.cache-size: 1GB
>>>>>>>>>>> performance.readdir-ahead: on
>>>>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Using simple dd test on VM in ovirt:
>>>>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G
count=1
>>>>>>>>>>> oflag=direct
>>>>>>>>> block size of 1G?!
>>>>>>>>>
>>>>>>>>> Try 1M (our default for storage operations)
>>>>>>>>>
>>>>>>>>>>> 1073741824 bytes (1.1 GB) copied, 65.9337 s,
16.3 MB/s
>>>>>>>>>>>
>>>>>>>>>>> Another VM not in ovirt using nfs:
>>>>>>>>>>> dd if=/dev/zero of=/root/testfile bs=1G
count=1
>>>>>>>>>>> oflag=direct
>>>>>>>>>>> 1073741824 bytes (1.1 GB) copied,
27.0079 s, 39.8 MB/s
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Is that expected or is there a better way to
set it up to
>>>>>>>>>>> get better
>>>>>>>>>>> performance?
>>>>>>>>> Adding Niels for advice.
>>>>>>>>>
>>>>>>>>>>> This email, its contents and ....
>>>>>>>>> Please avoid this, this is a public mailing list,
everything
>>>>>>>>> you write
>>>>>>>>> here is public.
>>>>>>>>>
>>>>>>>>> Nir
>>>>>>>> I'll have to look into how to remove this sig for
this mailing
>>>>>>>> list....
>>>>>>>>
>>>>>>>> Cloud Services for Business www.j2.com
>>>>>>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe |
Onebox
>>>>>>>>
>>>>>>>>
>>>>>>>> This email, its contents and attachments contain
information
>>>>>>>> from j2 Global,
>>>>>>>> Inc. and/or its affiliates which may be privileged,
>>>>>>>> confidential or
>>>>>>>> otherwise protected from disclosure. The information is
>>>>>>>> intended to be for
>>>>>>>> the addressee(s) only. If you are not an addressee, any
>>>>>>>> disclosure, copy,
>>>>>>>> distribution, or use of the contents of this message is
>>>>>>>> prohibited. If you
>>>>>>>> have received this email in error please notify the
sender by
>>>>>>>> reply e-mail
>>>>>>>> and delete the original message and any copies. (c) 2015
j2
>>>>>>>> Global, Inc. All
>>>>>>>> rights reserved. eFax, eVoice, Campaigner, FuseMail,
>>>>>>>> KeepItSafe, and Onebox
>>>>>>>> are registered trademarks of j2 Global, Inc. and its
affiliates.
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users(a)ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>> www.j2.com
>>>>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>>>>
>>>>> This email, its contents and attachments contain information from
>>>>> j2 Global, Inc
>>>>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
>>>>> and/or its affiliates which may be privileged, confidential or
>>>>> otherwise protected from disclosure. The information is intended
>>>>> to be for the addressee(s) only. If you are not an addressee, any
>>>>> disclosure, copy, distribution, or use of the contents of this
>>>>> message is prohibited. If you have received this email in error
>>>>> please notify the sender by reply e-mail and delete the original
>>>>> message and any copies. © 2015 j2 Global, Inc
>>>>> <http://www.j2.com/>;. All rights reserved. eFax ®
>>>>> <http://www.efax.com/>;, eVoice ®
<http://www.evoice.com/>;,
>>>>> Campaigner ® <http://www.campaigner.com/>;, FuseMail ®
>>>>> <http://www.fusemail.com/>;, KeepItSafe ®
>>>>> <http://www.keepitsafe.com/> and Onebox ®
>>>>> <http://www.onebox.com/> are r egistered trademarks of j2
Global,
>>>>> Inc <http://www.j2.com/>;. and its affiliates.
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>> www.j2.com
>>>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>>>
>>>> This email, its contents and attachments contain information from
>>>> j2 Global, Inc
>>>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.
>>>> and/or its affiliates which may be privileged, confidential or
>>>> otherwise protected from disclosure. The information is intended
>>>> to be for the addressee(s) only. If you are not an addressee, any
>>>> disclosure, copy, distribution, or use of the contents of this
>>>> message is prohibited. If you have received this email in error
>>>> please notify the sender by reply e-mail and delete the original
>>>> message and any copies. © 2015 j2 Global, Inc
>>>> <http://www.j2.com/>;. All rights reserved. eFax ®
>>>> <http://www.efax.com/>;, eVoice ® <http://www.evoice.com/>;,
>>>> Campaigner ® <http://www.campaigner.com/>;, FuseMail ®
>>>> <http://www.fusemail.com/>;, KeepItSafe ®
>>>> <http://www.keepitsafe.com/> and Onebox ®
<http://www.onebox.com/>
>>>> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;.
>>>> and its affiliates.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>>> --
>>> This message is subject to the CSIR's copyright terms and
>>> conditions, e-mail legal notice, and implemented Open Document
>>> Format (ODF) standard.
>>> The full disclaimer details can be found at
>>> http://www.csir.co.za/disclaimer.html.
>>>
>>>
>>> This message has been scanned for viruses and dangerous content by
>>> *MailScanner* <http://www.mailscanner.info/>;,
>>> and is believed to be clean.
>>>
>>>
>>> Please consider the environment before printing this email.
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>> --
>> This message is subject to the CSIR's copyright terms and
>> conditions, e-mail legal notice, and implemented Open Document
>> Format (ODF) standard.
>> The full disclaimer details can be found at
>> http://www.csir.co.za/disclaimer.html.
>>
>>
>> This message has been scanned for viruses and dangerous content by
>> *MailScanner* <http://www.mailscanner.info/>;,
>> and is believed to be clean.
>>
>>
>> Please consider the environment before printing this email.
>>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
--
This message is subject to the CSIR's copyright terms and conditions,
e-mail legal notice, and implemented Open Document Format (ODF) standard.
The full disclaimer details can be found at
http://www.csir.co.za/disclaimer.html.
This message has been scanned for viruses and dangerous content by
*MailScanner* <http://www.mailscanner.info/>;,
and is believed to be clean.
Please consider the environment before printing this email.
--------------020300020609000301040107
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 04/12/2016 02:41 PM, Roderick Mooi
wrote:<br>
</div>
<blockquote
cite="mid:1438EFA3-D224-4339-928E-04F2A5BEC128@csir.co.za"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hi
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div text="#000000" bgcolor="#FFFFFF"
class="">It is not
removed. Can you try 'gluster volume set volname
cluster.eager-lock enable`?</div>
</blockquote>
<div class=""><br class="">
</div>
This works. BTW by default this setting is on. Whats the
difference between on and enable?</div>
</blockquote>
<br>
Both are identical. You can use any of the booleans to achieve the
same effect. {"1", "on", "yes", "true",
"enable"} or {"0", "off",
"no", "false", "disable"}<br>
FYI, the patch <a class="moz-txt-link-freetext"
href="http://review.gluster.org/#/c/13958/">http://review.gl...
to fix this
issue should make it to glusterfs-3.7.11.<br>
-Ravi<br>
<blockquote
cite="mid:1438EFA3-D224-4339-928E-04F2A5BEC128@csir.co.za"
type="cite">
<div class=""><br class="">
</div>
<div class="">Thanks for the clarification.</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class=""><br class="">
</div>
<div class="">Roderick</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 06 Apr 2016, at 10:56 AM, Ravishankar N
<<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
class="">ravishankar(a)redhat.com</a>&gt;
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF"
class="">
<div class="moz-cite-prefix">On 04/06/2016 02:08 PM,
Roderick Mooi wrote:<br class="">
</div>
<blockquote
cite="mid:90A2411F-0F73-437D-899B-6098A5C7A1B6@csir.co.za"
type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
Hi Ravi and colleagues
<div class=""><br class="">
</div>
<div class="">(apologies for hijacking this thread but
Im not sure where else to report this (and it is
related).)
<div class=""><br class="">
</div>
<div class="">With gluster 3.7.10,
running</div>
<div class="">#gluster volume set
<volname>
group virt</div>
<div class="">fails with:</div>
<div class="">volume set: failed: option :
eager-lock does not exist</div>
<div class="">Did you mean eager-lock?</div>
<div class=""><br class="">
</div>
<div class="">I had to remove the eager-lock setting
from /var/lib/glusterd/groups/virt to get this to
work. It seems like setting eager-lock has been
removed from latest gluster. Is this correct?
Either way, is there anything else I should do?</div>
</div>
</blockquote>
<br class="">
It is not removed. Can you try 'gluster volume set
volname cluster.eager-lock enable`?<br class="">
I think the disperse (EC) translator introduced a
`disperse.eager-lock` which is why you would need to
mention entire volume option name to avoid ambiguity.<br
class="">
We probably need to fix the virt profile setting to
include the entire name. By the way 'gluster volume set
help` should give you the list of all options.<br
class="">
<br class="">
-Ravi<br class="">
<br class="">
<blockquote
cite="mid:90A2411F-0F73-437D-899B-6098A5C7A1B6@csir.co.za"
type="cite" class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">Roderick<br class="">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 12 Feb 2016, at 6:18 AM,
Ravishankar N <<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
class=""><a
class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a></a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html;
charset=windows-1252"
http-equiv="Content-Type"
class="">
<div text="#000000"
bgcolor="#FFFFFF"
class="">
<div class="moz-cite-prefix">Hi
Bill,<br
class="">
Can you enable virt-profile setting
for your volume and see if that helps?
You need to enable this optimization
when you create the volume using
ovrit, or use the following command
for an existing volume:<br class="">
<tt class=""><br
class="">
#gluster volume set <volname>
group virt</tt><br class="">
<br class="">
-Ravi<br class="">
<br class="">
<br class="">
On 02/12/2016 05:22 AM, Bill James
wrote:<br class="">
</div>
<blockquote
cite="mid:56BD1EB9.5000802@j2.com"
type="cite" class="">
<meta content="text/html;
charset=windows-1252"
http-equiv="Content-Type"
class="">
My apologies, I'm showing how much of
a noob I am.<br class="">
Ignore last direct to gluster numbers,
as that wasn't really glusterfs.<br
class="">
<br class="">
<br class="">
[root@ovirt2 test ~]# mount -t
glusterfs <a moz-do-not-send="true"
href="http://ovirt2-ks.test.j2noc.com/"
class="">ovirt2-ks.test.j2noc.com</a>:/gv1
/mnt/tmp/<br class="">
[root@ovirt2 test ~]# time dd
if=/dev/zero of=/mnt/tmp/testfile2
bs=1M count=1000 oflag=direct<br
class="">
1048576000 bytes (1.0 GB) copied,
65.8596 s, 15.9 MB/s<br class="">
<br class="">
That's more how I expected, it is
pointing to glusterfs performance.<br
class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On
02/11/2016 03:27 PM, Bill James
wrote:<br class="">
</div>
<blockquote
cite="mid:56BD18D7.8080909@j2.com"
type="cite" class="">
<meta http-equiv="Content-Type"
content="text/html;
charset=windows-1252" class="">
don't know if it helps, but I ran a
few more tests, all from the same
hardware node.<br class="">
<br class="">
The VM:<br class="">
[root@billjov1 ~]# time dd
if=/dev/zero of=/root/testfile bs=1M
count=1000 oflag=direct<br class="">
1048576000 bytes (1.0 GB) copied,
62.5535 s, 16.8 MB/s<br class="">
<br class="">
Writing directly to gluster volume:<br
class="">
[root@ovirt2 test ~]# time dd
if=/dev/zero
of=/gluster-store/brick1/gv1/testfile
bs=1M count=1000 oflag=direct<br
class="">
1048576000 bytes (1.0 GB) copied,
9.92048 s, 106 MB/s<br class="">
<br class="">
<br class="">
Writing to NFS volume:<br class="">
[root@ovirt2 test ~]# time dd
if=/dev/zero
of=/mnt/storage/qa/testfile bs=1M
count=1000 oflag=direct<br class="">
1048576000 bytes (1.0 GB) copied,
10.5776 s, 99.1 MB/s<br class="">
<br class="">
NFS & Gluster are using the same
interface. Tests were not run at
same time.<br class="">
<br class="">
This would suggest my problem isn't
glusterfs, but the VM performance.<br
class="">
<br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On
02/11/2016 03:13 PM, Bill James
wrote:<br class="">
</div>
<blockquote
cite="mid:56BD1589.8090506@j2.com"
type="cite" class="">xml
attached.
<br class="">
<br class="">
<br class="">
On 02/11/2016 12:28 PM, Nir Soffer
wrote: <br class="">
<blockquote type="cite"
class="">On
Thu, Feb 11, 2016 at 8:27 PM,
Bill James <a
moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><a class="moz-txt-link-rfc2396E"
href="mailto:bill.james@j2.com"><bill.james@j2.com></a></a>
wrote: <br
class="">
<blockquote type="cite"
class="">thank
you for the reply. <br
class="">
<br class="">
We setup gluster using the
names associated with NIC 2
IP. <br class="">
Brick1: <a
moz-do-not-send="true"
href="http://ovirt1-ks.test.j2noc.com/"
class="">ovirt1-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick2: <a
moz-do-not-send="true"
href="http://ovirt2-ks.test.j2noc.com/"
class="">ovirt2-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick3: <a
moz-do-not-send="true"
href="http://ovirt3-ks.test.j2noc.com/"
class="">ovirt3-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
<br class="">
That's NIC 2's IP. <br
class="">
Using 'iftop -i eno2 -L 5 -t'
: <br class="">
<br class="">
dd if=/dev/zero
of=/root/testfile bs=1M
count=1000 oflag=direct <br
class="">
1048576000 bytes (1.0 GB)
copied, 68.0714 s, 15.4 MB/s <br
class="">
</blockquote>
Can you share the xml of this
vm? You can find it in vdsm log,
<br class="">
at the time you start the vm. <br
class="">
<br class="">
Or you can do (on the host): <br
class="">
<br class="">
# virsh <br class="">
virsh # list <br class="">
(username: vdsm@ovirt password:
shibboleth) <br class="">
virsh # dumpxml vm-id <br
class="">
<br class="">
<blockquote type="cite"
class="">Peak
rate
(sent/received/total):
281Mb 5.36Mb <br class="">
282Mb <br class="">
Cumulative
(sent/received/total):
1.96GB 14.6MB <br class="">
1.97GB <br class="">
<br class="">
gluster volume info gv1: <br
class="">
Options Reconfigured: <br
class="">
performance.write-behind-window-size:
4MB <br class="">
performance.readdir-ahead:
on <br class="">
performance.cache-size: 1GB
<br class="">
performance.write-behind:
off <br class="">
<br class="">
performance.write-behind: off
didn't help. <br class="">
Neither did any other changes
I've tried. <br class="">
<br class="">
<br class="">
There is no VM traffic on this
VM right now except my test. <br
class="">
<br class="">
<br class="">
<br class="">
On 02/10/2016 11:55 PM, Nir
Soffer wrote: <br class="">
<blockquote type="cite"
class="">On Thu, Feb 11,
2016 at 2:42 AM, Ravishankar
N <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a>
<br class="">
wrote: <br class="">
<blockquote type="cite"
class="">+gluster-users <br
class="">
<br class="">
Does disabling
'performance.write-behind'
give a better throughput?
<br class="">
<br class="">
<br class="">
<br class="">
On 02/10/2016 11:06 PM,
Bill James wrote: <br
class="">
<blockquote type="cite"
class="">I'm setting up
a ovirt cluster using
glusterfs and noticing
not stellar <br
class="">
performance. <br
class="">
Maybe my setup could use
some adjustments? <br
class="">
<br class="">
3 hardware nodes running
centos7.2, glusterfs
3.7.6.1, ovirt
3.6.2.6-1. <br class="">
Each node has 8 spindles
configured in 1 array
which is split using LVM
<br class="">
with one logical volume
for system and one for
gluster. <br class="">
They each have 4 NICs, <br
class="">
NIC1 = ovirtmgmt <br
class="">
NIC2 = gluster
(1GbE) <br class="">
</blockquote>
</blockquote>
How do you ensure that
gluster trafic is using this
nic? <br class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class=""> NIC3 = VM
traffic <br class="">
</blockquote>
</blockquote>
How do you ensure that vm
trafic is using this nic? <br
class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">I tried with
default glusterfs
settings <br class="">
</blockquote>
</blockquote>
And did you find any
difference? <br class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">and also with:
<br class="">
performance.cache-size:
1GB <br class="">
performance.readdir-ahead:
on <br class="">
performance.write-behind-window-size:
4MB <br class="">
<br class="">
[root@ovirt3 test
scripts]# gluster volume
info gv1 <br class="">
<br class="">
Volume Name: gv1 <br
class="">
Type: Replicate <br
class="">
Volume ID:
71afc35b-09d7-4384-ab22-57d032a0f1a2
<br class="">
Status: Started <br
class="">
Number of Bricks: 1 x 3
= 3 <br class="">
Transport-type: tcp <br
class="">
Bricks: <br class="">
Brick1: <a
moz-do-not-send="true"
href="http://ovirt1-ks.test.j2noc.com/"
class="">ovirt1-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick2: <a
moz-do-not-send="true"
href="http://ovirt2-ks.test.j2noc.com/"
class="">ovirt2-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Brick3: <a
moz-do-not-send="true"
href="http://ovirt3-ks.test.j2noc.com/"
class="">ovirt3-ks.test.j2noc.com</a>:/gluster-store/brick1/gv1
<br class="">
Options Reconfigured: <br
class="">
performance.cache-size:
1GB <br class="">
performance.readdir-ahead:
on <br class="">
performance.write-behind-window-size:
4MB <br class="">
<br class="">
<br class="">
Using simple dd test on
VM in ovirt: <br
class="">
dd if=/dev/zero
of=/root/testfile bs=1G
count=1 oflag=direct <br
class="">
</blockquote>
</blockquote>
block size of 1G?! <br
class="">
<br class="">
Try 1M (our default for
storage operations) <br
class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">
1073741824 bytes (1.1
GB) copied, 65.9337 s,
16.3 MB/s <br class="">
<br class="">
Another VM not in ovirt
using nfs: <br class="">
dd if=/dev/zero
of=/root/testfile bs=1G
count=1 oflag=direct <br
class="">
1073741824 bytes
(1.1 GB) copied, 27.0079
s, 39.8 MB/s <br
class="">
<br class="">
<br class="">
Is that expected or is
there a better way to
set it up to get better
<br class="">
performance? <br
class="">
</blockquote>
</blockquote>
Adding Niels for advice. <br
class="">
<br class="">
<blockquote type="cite"
class="">
<blockquote type="cite"
class="">This email, its
contents and .... <br
class="">
</blockquote>
</blockquote>
Please avoid this, this is a
public mailing list,
everything you write <br
class="">
here is public. <br
class="">
<br class="">
Nir <br class="">
</blockquote>
I'll have to look into how to
remove this sig for this
mailing list.... <br class="">
<br class="">
Cloud Services for Business <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="http://www.j2.com/"><a
class="moz-txt-link-abbreviated"
href="http://www.j2.com">www.j2.com</a></a>
<br class="">
j2 | eFax | eVoice | FuseMail
| Campaigner | KeepItSafe |
Onebox <br class="">
<br class="">
<br class="">
This email, its contents and
attachments contain
information from j2 Global, <br
class="">
Inc. and/or its affiliates
which may be privileged,
confidential or <br class="">
otherwise protected from
disclosure. The information is
intended to be for <br
class="">
the addressee(s) only. If you
are not an addressee, any
disclosure, copy, <br
class="">
distribution, or use of the
contents of this message is
prohibited. If you <br
class="">
have received this email in
error please notify the sender
by reply e-mail <br class="">
and delete the original
message and any copies. (c)
2015 j2 Global, Inc. All <br
class="">
rights reserved. eFax, eVoice,
Campaigner, FuseMail,
KeepItSafe, and Onebox <br
class="">
are registered trademarks of
j2 Global, Inc. and its
affiliates. <br class="">
</blockquote>
</blockquote>
<br class="">
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
<p class=""><a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class=""><span
style="color:windowtext;
text-decoration:none"
class=""><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" class=""
height="46" width="391"
border="0"></span></a></p>
<p class=""><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray"
class="">This email, its
contents and attachments contain
information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class="">j2 Global, Inc</a>.
and/or its affiliates which may
be privileged, confidential or
otherwise protected from
disclosure. The information is
intended to be for the
addressee(s) only. If you are
not an addressee, any
disclosure, copy, distribution,
or use of the contents of this
message is prohibited. If you
have received this email in
error please notify the sender
by reply e-mail and delete the
original message and any copies.
© 2015 <a
moz-do-not-send="true"
href="http://www.j2.com/"
class="">j2 Global, Inc</a>.
All rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/"
class="">eFax ®</a>, <a
moz-do-not-send="true"
href="http://www.evoice.com/"
class="">eVoice ®</a>, <a
moz-do-not-send="true"
href="http://www.campaigner.com/"
class="">Campaigner ®</a>,
<a
moz-do-not-send="true"
href="http://www.fusemail.com/"
class="">FuseMail ®</a>,
<a
moz-do-not-send="true"
href="http://www.keepitsafe.com/"
class="">KeepItSafe ®</a>
and
<a moz-do-not-send="true"
href="http://www.onebox.com/"
class="">Onebox ®</a> are r
egistered trademarks of <a
moz-do-not-send="true"
href="http://www.j2.com/"
class="">j2 Global, Inc</a>.
and its affiliates.</span></p>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
<p class=""><a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class=""><span
style="color:windowtext;
text-decoration:none"
class=""><img
moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
alt="www.j2.com" class=""
height="46" width="391"
border="0"></span></a></p>
<p class=""><span
style="font-size:8.0pt;font-family:"Arial","sans-serif";color:gray"
class="">This email, its contents
and attachments contain
information from <a
moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsel...
class="">j2 Global, Inc</a>.
and/or its affiliates which may be
privileged, confidential or
otherwise protected from
disclosure. The information is
intended to be for the
addressee(s) only. If you are not
an addressee, any disclosure,
copy, distribution, or use of the
contents of this message is
prohibited. If you have received
this email in error please notify
the sender by reply e-mail and
delete the original message and
any copies. © 2015 <a
moz-do-not-send="true"
href="http://www.j2.com/"
class="">j2 Global, Inc</a>.
All
rights reserved. <a
moz-do-not-send="true"
href="http://www.efax.com/"
class="">eFax ®</a>, <a
moz-do-not-send="true"
href="http://www.evoice.com/"
class="">eVoice ®</a>, <a
moz-do-not-send="true"
href="http://www.campaigner.com/"
class="">Campaigner ®</a>,
<a
moz-do-not-send="true"
href="http://www.fusemail.com/"
class="">FuseMail ®</a>, <a
moz-do-not-send="true"
href="http://www.keepitsafe.com/"
class="">KeepItSafe ®</a> and
<a
moz-do-not-send="true"
href="http://www.onebox.com/"
class="">Onebox ®</a> are r
egistered trademarks of <a
moz-do-not-send="true"
href="http://www.j2.com/"
class="">j2 Global, Inc</a>.
and
its affiliates.</span></p>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class=""
wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br class="">
<br class="">
<font class=""
face="Verdana,Arial,Helvetica,Trebuchet
MS" size="1"> <br
class="">
-- <br class="">
This message is subject to the CSIR's
copyright terms and conditions, e-mail
legal notice, and implemented Open
Document Format (ODF) standard. <br
class="">
The full disclaimer details can be
found at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://www.csir.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a>.
<p class=""> <br
class="">
This message has been scanned for
viruses and dangerous content by <a
moz-do-not-send="true"
href="http://www.mailscanner.info/"
class=""><b
class="">MailScanner</b></a>,
<br class="">
and is believed to be clean. </p>
</font>
<p class=""><font class=""
face="Verdana,Arial,Helvetica,Trebuchet
MS" size="1"> <br
class="">
Please consider the environment
before printing this email. </font>
</p>
</div>
_______________________________________________<br class="">
Users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
class="">Users(a)ovirt.org</a><br
class="">
<a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
<font class=""
face="Verdana,Arial,Helvetica,Trebuchet
MS" size="1"> <br class="">
-- <br class="">
This message is subject to the CSIR's copyright
terms and conditions, e-mail legal notice, and
implemented Open Document Format (ODF) standard. <br
class="">
The full disclaimer details can be found at <a
moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://www.csir.co.za/disclaimer.html"><a
class="moz-txt-link-freetext"
href="http://www.csir.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a></a>.
<p class=""> <br class="">
This message has been scanned for viruses and
dangerous content by <a moz-do-not-send="true"
href="http://www.mailscanner.info/"
class=""><b
class="">MailScanner</b></a>, <br
class="">
and is believed to be clean. </p>
</font>
<p class=""><font class=""
face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1"> <br class="">
Please consider the environment before printing
this email. </font> </p>
</blockquote>
<br class="">
<br class="">
</div>
_______________________________________________<br
class="">
Users mailing list<br class="">
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
class="">Users(a)ovirt.org</a><br
class="">
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<font face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1">
<br>
-- <br>
This message is subject to the CSIR's copyright terms and
conditions, e-mail legal notice, and implemented Open Document
Format (ODF) standard.
<br>
The full disclaimer details can be found at <a
moz-do-not-send="true"
href="http://www.csir.co.za/disclaimer.html"><a
class="moz-txt-link-freetext"
href="http://www.csir.co.za/disclaimer.html">http://www.csir.co.za/disclaimer.html</a></a>.
<p>
<br>
This message has been scanned for viruses and dangerous
content by <a moz-do-not-send="true"
href="http://www.mailscanner.info/"><b>MailScanner</b></a>,
<br>
and is believed to be clean.
</p>
</font>
<p><font face="Verdana,Arial,Helvetica,Trebuchet MS"
size="1">
<br>
Please consider the environment before printing this email.
</font>
</p>
</blockquote>
<br>
<br>
</body>
</html>
--------------020300020609000301040107--
12:59 p.m.
--PW0Eas8rCkcu1VkF
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, Apr 12, 2016 at 11:11:54AM +0200, Roderick Mooi wrote:
Hi
=20
> It is not removed. Can you try 'gluster volume set volname cluster.eage=
r-lock enable`?
=20
This works. BTW by default this setting is =E2=80=9Con=E2=80=9D
Thanks for reporting back!
What=E2=80=99s the difference between =E2=80=9Con=E2=80=9D and
=E2=80=9Ce=
nable=E2=80=9D?
Both are the same, you could also use "yes", "true" and possibly
others.
Cheers,
Niels
=20
Thanks for the clarification.
=20
Regards,
=20
Roderick
=20
> On 06 Apr 2016, at 10:56 AM, Ravishankar N <ravishankar(a)redhat.com> wro=
te:
>=20
> On 04/06/2016 02:08 PM, Roderick Mooi wrote:
>> Hi Ravi and colleagues
>>=20
>> (apologies for hijacking this thread but I=E2=80=99m not sure where el=
se
to report this (and it is related).)
>>=20
>> With gluster 3.7.10, running
>> #gluster volume set <volname> group virt
>> fails with:
>> volume set: failed: option : eager-lock does not exist
>> Did you mean eager-lock?
>>=20
>> I had to remove the eager-lock setting from /var/lib/glusterd/groups/v=
irt to get this to work. It seems like setting eager-lock has been removed =
=66rom latest gluster. Is this correct? Either way, is there anything else =
I should do?
>=20
> It is not removed. Can you try 'gluster volume set volname cluster.eage=
r-lock enable`?
> I think the disperse (EC) translator introduced a
`disperse.eager-lock`=
which is why you would need to mention entire volume option
name to avoid =
ambiguity.
> We probably need to fix the virt profile setting to include the
entire =
name. By the way 'gluster volume set help` should give you the list of
all =
options.
>=20
> -Ravi
>=20
>>=20
>> Cheers,
>>=20
>> Roderick
>>=20
>>> On 12 Feb 2016, at 6:18 AM, Ravishankar N <ravishankar(a)redhat.com <ma=
ilto:ravishankar@redhat.com>> wrote:
>>>=20
>>> Hi Bill,
>>> Can you enable virt-profile setting for your volume and see if that h=
elps? You need to enable this optimization when you create the volume using=
ovrit, or use the following command for an existing volume:
>>>=20
>>> #gluster volume set <volname> group virt
>>>=20
>>> -Ravi
>>>=20
>>>=20
>>> On 02/12/2016 05:22 AM, Bill James wrote:
>>>> My apologies, I'm showing how much of a noob I am.
>>>> Ignore last direct to gluster numbers, as that wasn't really
gluster=
fs.
>>>>=20
>>>>=20
>>>> [root@ovirt2 test ~]# mount -t glusterfs ovirt2-ks.test.j2noc.com <h=
ttp://ovirt2-ks.test.j2noc.com/>:/gv1 /mnt/tmp/
>>>> [root@ovirt2 test ~]# time dd if=3D/dev/zero
of=3D/mnt/tmp/testfile2=
bs=3D1M count=3D1000 oflag=3Ddirect
>>>> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9
MB/s
>>>>=20
>>>> That's more how I expected, it is pointing to glusterfs
performance.
>>>>=20
>>>>=20
>>>>=20
>>>> On 02/11/2016 03:27 PM, Bill James wrote:
>>>>> don't know if it helps, but I ran a few more tests, all from the
sa=
me hardware node.
>>>>>=20
>>>>> The VM:
>>>>> [root@billjov1 ~]# time dd if=3D/dev/zero of=3D/root/testfile bs=3D=
1M count=3D1000 oflag=3Ddirect
>>>>> 1048576000 bytes (1.0 GB) copied, 62.5535 s,
16.8 MB/s
>>>>>=20
>>>>> Writing directly to gluster volume:
>>>>> [root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/gluster-store/br=
ick1/gv1/testfile bs=3D1M count=3D1000 oflag=3Ddirect
>>>>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106
MB/s
>>>>>=20
>>>>>=20
>>>>> Writing to NFS volume:
>>>>> [root@ovirt2 test ~]# time dd if=3D/dev/zero of=3D/mnt/storage/qa/t=
estfile bs=3D1M count=3D1000 oflag=3Ddirect
>>>>> 1048576000 bytes (1.0 GB) copied, 10.5776 s,
99.1 MB/s
>>>>>=20
>>>>> NFS & Gluster are using the same interface. Tests were not run
at s=
ame time.
>>>>>=20
>>>>> This would suggest my problem isn't glusterfs, but the VM
performan=
ce.
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> On 02/11/2016 03:13 PM, Bill James wrote:
>>>>>> xml attached.=20
>>>>>>=20
>>>>>>=20
>>>>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:=20
>>>>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James
<mailto:bill.james@j=
2.com><bill.james(a)j2.com>
<mailto:bill.james@j2.com> wrote:=20
>>>>>>>> thank you for the reply.=20
>>>>>>>>=20
>>>>>>>> We setup gluster using the names associated with NIC 2
IP.=20
>>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
<http://ovirt1-ks.test.j2noc.=
com/>:/gluster-store/brick1/gv1=20
>>>>>>>> Brick2: ovirt2-ks.test.j2noc.com
<http://ovirt2-ks.test.j2noc.=
com/>:/gluster-store/brick1/gv1=20
>>>>>>>> Brick3: ovirt3-ks.test.j2noc.com
<http://ovirt3-ks.test.j2noc.=
com/>:/gluster-store/brick1/gv1=20
>>>>>>>>=20
>>>>>>>> That's NIC 2's IP.=20
>>>>>>>> Using 'iftop -i eno2 -L 5 -t' :=20
>>>>>>>>=20
>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile bs=3D1M
count=3D1000 oflag=
=3Ddirect=20
>>>>>>>> 1048576000 bytes (1.0 GB) copied,
68.0714 s, 15.4 MB/s=20
>>>>>>> Can you share the xml of this vm? You can find it in vdsm
log,=20
>>>>>>> at the time you start the vm.=20
>>>>>>>=20
>>>>>>> Or you can do (on the host):=20
>>>>>>>=20
>>>>>>> # virsh=20
>>>>>>> virsh # list=20
>>>>>>> (username: vdsm@ovirt password: shibboleth)=20
>>>>>>> virsh # dumpxml vm-id=20
>>>>>>>=20
>>>>>>>> Peak rate (sent/received/total):
281Mb 5.36=
Mb=20
>>>>>>>> 282Mb=20
>>>>>>>> Cumulative (sent/received/total):
1.96GB 14.6=
MB=20
>>>>>>>> 1.97GB=20
>>>>>>>>=20
>>>>>>>> gluster volume info gv1:=20
>>>>>>>> Options Reconfigured:=20
>>>>>>>> performance.write-behind-window-size: 4MB=20
>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>> performance.write-behind: off=20
>>>>>>>>=20
>>>>>>>> performance.write-behind: off didn't help.=20
>>>>>>>> Neither did any other changes I've tried.=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> There is no VM traffic on this VM right now except my
test.=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:=20
>>>>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N
<mailto:ravisha=
nkar@redhat.com><ravishankar(a)redhat.com>
<mailto:ravishankar@redhat.com>=20
>>>>>>>>> wrote:=20
>>>>>>>>>> +gluster-users=20
>>>>>>>>>>=20
>>>>>>>>>> Does disabling
'performance.write-behind' give a better throug=
hput?=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:=20
>>>>>>>>>>> I'm setting up a ovirt cluster using
glusterfs and noticing n=
ot stellar=20
>>>>>>>>>>> performance.=20
>>>>>>>>>>> Maybe my setup could use some
adjustments?=20
>>>>>>>>>>>=20
>>>>>>>>>>> 3 hardware nodes running centos7.2,
glusterfs 3.7.6.1, ovirt =
3.6.2.6-1.=20
>>>>>>>>>>> Each node has 8 spindles
configured in 1 array which is split=
using LVM=20
>>>>>>>>>>> with one logical volume
for system and one for gluster.=20
>>>>>>>>>>> They each have 4 NICs,=20
>>>>>>>>>>> NIC1 =3D ovirtmgmt=20
>>>>>>>>>>> NIC2 =3D gluster (1GbE)=20
>>>>>>>>> How do you ensure that gluster trafic is using this
nic?=20
>>>>>>>>>=20
>>>>>>>>>>> NIC3 =3D VM traffic=20
>>>>>>>>> How do you ensure that vm trafic is using this
nic?=20
>>>>>>>>>=20
>>>>>>>>>>> I tried with default glusterfs settings=20
>>>>>>>>> And did you find any difference?=20
>>>>>>>>>=20
>>>>>>>>>>> and also with:=20
>>>>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>>>>> performance.write-behind-window-size:
4MB=20
>>>>>>>>>>>=20
>>>>>>>>>>> [root@ovirt3 test scripts]# gluster volume
info gv1=20
>>>>>>>>>>>=20
>>>>>>>>>>> Volume Name: gv1=20
>>>>>>>>>>> Type: Replicate=20
>>>>>>>>>>> Volume ID:
71afc35b-09d7-4384-ab22-57d032a0f1a2=20
>>>>>>>>>>> Status: Started=20
>>>>>>>>>>> Number of Bricks: 1 x 3 =3D 3=20
>>>>>>>>>>> Transport-type: tcp=20
>>>>>>>>>>> Bricks:=20
>>>>>>>>>>> Brick1: ovirt1-ks.test.j2noc.com
<http://ovirt1-ks.test.j2noc=
=2Ecom/>:/gluster-store/brick1/gv1=20
>>>>>>>>>>> Brick2:
ovirt2-ks.test.j2noc.com <http://ovirt2-ks.test.j2noc=
=2Ecom/>:/gluster-store/brick1/gv1=20
>>>>>>>>>>> Brick3:
ovirt3-ks.test.j2noc.com <http://ovirt3-ks.test.j2noc=
=2Ecom/>:/gluster-store/brick1/gv1=20
>>>>>>>>>>> Options
Reconfigured:=20
>>>>>>>>>>> performance.cache-size: 1GB=20
>>>>>>>>>>> performance.readdir-ahead: on=20
>>>>>>>>>>> performance.write-behind-window-size:
4MB=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> Using simple dd test on VM in ovirt:=20
>>>>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile
bs=3D1G count=3D1 o=
flag=3Ddirect=20
>>>>>>>>> block size of 1G?!=20
>>>>>>>>>=20
>>>>>>>>> Try 1M (our default for storage operations)=20
>>>>>>>>>=20
>>>>>>>>>>> 1073741824 bytes (1.1 GB) copied,
65.9337 s, 16.3 MB/=
s=20
>>>>>>>>>>>=20
>>>>>>>>>>> Another VM not in ovirt using nfs:=20
>>>>>>>>>>> dd if=3D/dev/zero of=3D/root/testfile
bs=3D1G count=3D1 =
oflag=3Ddirect=20
>>>>>>>>>>> 1073741824 bytes
(1.1 GB) copied, 27.0079 s, 39.8 MB/=
s=20
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> Is that expected or is there a better way to
set it up to get=
better=20
>>>>>>>>>>> performance?=20
>>>>>>>>> Adding Niels for advice.=20
>>>>>>>>>=20
>>>>>>>>>>> This email, its contents and ....=20
>>>>>>>>> Please avoid this, this is a public mailing list,
everything yo=
u write=20
>>>>>>>>> here is public.=20
>>>>>>>>>=20
>>>>>>>>> Nir=20
>>>>>>>> I'll have to look into how to remove this sig for
this mailing l=
ist....=20
>>>>>>>>=20
>>>>>>>> Cloud Services for Business
<http://www.j2.com/>www.j2.com <htt=
p://www.j2.com/>=20
>>>>>>>> j2 | eFax | eVoice | FuseMail |
Campaigner | KeepItSafe | Onebox=
=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> This email, its contents and attachments contain
information fro=
m j2 Global,=20
>>>>>>>> Inc. and/or its affiliates which may
be privileged, confidential=
or=20
>>>>>>>> otherwise protected from disclosure.
The information is intended=
to be for=20
>>>>>>>> the addressee(s) only. If you are
not an addressee, any disclosu=
re, copy,=20
>>>>>>>> distribution, or use of the contents
of this message is prohibit=
ed. If you=20
>>>>>>>> have received this email in error
please notify the sender by re=
ply e-mail=20
>>>>>>>> and delete the original message and
any copies. (c) 2015 j2 Glob=
al, Inc. All=20
>>>>>>>> rights reserved. eFax, eVoice,
Campaigner, FuseMail, KeepItSafe,=
and Onebox=20
>>>>>>>> are registered trademarks of j2
Global, Inc. and its affiliates.=
=20
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.=
org/mailman/listinfo/users>
ral&utm_campaign=3Demployeeemail>
>>>>> This email, its contents and attachments contain
information from j=
2 Global, Inc
<http://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-=
referral&utm_campaign=3Demployemail>. and/or its affiliates which may be pr=
ivileged, confidential or otherwise protected from disclosure. The informat=
ion is intended to be for the addressee(s) only. If you are not an addresse=
e, any disclosure, copy, distribution, or use of the contents of this messa=
ge is prohibited. If you have received this email in error please notify th=
e sender by reply e-mail and delete the original message and any copies. =
=C2=A9 2015 j2 Global, Inc <http://www.j2.com/>;. All rights reserved. eFax =
=C2=AE <http://www.efax.com/>;, eVoice =C2=AE <http://www.evoice.com/>;, Camp=
aigner =C2=AE <http://www.campaigner.com/>;, FuseMail =C2=AE <http://www.fus=
email.com/>;, KeepItSafe =C2=AE <http://www.keepitsafe.com/> and Onebox =C2=
=AE <http://www.onebox.com/> are r egistered trademarks of j2 Global, Inc <=
http://www.j2.com/>;. and its affiliates.
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>> http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.o=
rg/mailman/listinfo/users>
al&utm_campaign=3Demployeeemail>
>>>> This email, its contents and attachments contain
information from j2=
Global, Inc
<http://www.j2.com/?utm_source=3Dj2global&utm_medium=3Dxsell-r=
eferral&utm_campaign=3Demployemail>. and/or its affiliates which may be pri=
vileged, confidential or otherwise protected from disclosure. The informati=
on is intended to be for the addressee(s) only. If you are not an addressee=
, any disclosure, copy, distribution, or use of the contents of this messag=
e is prohibited. If you have received this email in error please notify the=
sender by reply e-mail and delete the original message and any copies. =C2=
=A9 2015 j2 Global, Inc <http://www.j2.com/>;. All rights reserved. eFax =C2=
=AE <http://www.efax.com/>;, eVoice =C2=AE <http://www.evoice.com/>;, Campaig=
ner =C2=AE <http://www.campaigner.com/>;, FuseMail =C2=AE <http://www.fusema=
il.com/>;, KeepItSafe =C2=AE <http://www.keepitsafe.com/> and Onebox =C2=AE =
<http://www.onebox.com/> are r egistered trademarks of j2 Global, Inc <http=
://www.j2.com/>. and its affiliates.
>>>>=20
>>>>=20
>>>>=20
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.or=
g/mailman/listinfo/users>
>>>=20
>>>=20
>>>=20
>>> --=20
>>> This message is subject to the CSIR's copyright terms and conditions,=
e-mail legal notice, and implemented Open Document Format (ODF) standard.=
=20
>>> The full disclaimer details can be found at
<http://www.csir.co.za/d=
isclaimer.html>http://www.csir.co.za/disclaimer.html
<http://www.csir.co.za=
/disclaimer.html>.
>>>=20
>>> This message has been scanned for viruses and dangerous content by Ma=
ilScanner <http://www.mailscanner.info/>,=20
>>> and is believed to be clean.
>>>=20
>>>=20
>>> Please consider the environment before printing this email.
>>>=20
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org=
/mailman/listinfo/users>
>>=20
>>=20
>> --=20
>> This message is subject to the CSIR's copyright terms and conditions, =
e-mail legal notice, and implemented Open Document Format (ODF) standard.=
=20
>> The full disclaimer details can be found at
<http://www.csir.co.za/di=
sclaimer.html>http://www.csir.co.za/disclaimer.html
<http://www.csir.co.za/=
disclaimer.html>.
>>=20
>> This message has been scanned for viruses and dangerous content by Mai=
lScanner <http://www.mailscanner.info/>,=20
>> and is believed to be clean.
>>=20
>>=20
>> Please consider the environment before printing this email.
>>=20
>=20
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
=20
=20
--=20
This message is subject to the CSIR's copyright terms and conditions, e-m=
ail
legal notice, and implemented Open Document Format (ODF) standard.=20
The full disclaimer details can be found at
http://www.csir.co.za/disclai=
mer.html.
=20
This message has been scanned for viruses and dangerous content by MailSc=
anner,=20
and is believed to be clean.
=20
Please consider the environment before printing this email.
=20
--PW0Eas8rCkcu1VkF
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBAgAGBQJXDMb9AAoJECXo5AApwsWzMjsQAMVh1jBmyG/6iYfL9V1grz9j
5i/fsIQB9kfEGeV5Jh8A1/o+DLnXLAwf8L0C1b4sW2nUQ//8nfrroDVrRaBqv/wL
vrzs5jcuzfTl56Zv6FGsYZDTrqVrACi997lKFkvzTiUOqH8hw9ctmVen+lzlCga7
yuYCb/SpeH/Nx5I5o2Gfnd54jd5FSdMZfqoCS6TVbVPtlFjbN/spBCT9VT4SpoIn
5RRGGbPqcd36BpWIyGp4iZ2BAtqfIZDcJWg2y+kbMFRWnIlJ5wkixtPvAlMIBC3d
ehyfjPRa2/aTIT24Xth/vDzAYL0S5phpnsvmlDiendxerz4eT+4PcP0TVws5NvD1
1tV1qNd4i7dnE8d7AlqjhnjEyMijIgjkF55BRj5PJ21CkTdwVsk5BcPmEP3lUH6c
HwwXZGPY5iC6Q+pJ0KERVutJDmp/OJ86SlTnRwinlHRTtjagklmcvkKyH44TgfaF
t+IE3/g6l0IgPkg4kU30ignUxaFvikH9Y6esnsHglzuVfAiMliR0hU7mLx0yrh5E
2fI80iV+alp7eRjkHEVZ7A/rKfOJWPY2y0+P6WnoKIkYQ+uNlZSOCERRJIv56XDO
66feEUuzxna/ucxBYyEiPmgNIx64PPaXr6eiA8cgG3TpLZgQ8E4Od6qP7MI+2ikR
W1Hy81Lq7KrQwGzgmZR8
=gQkK
-----END PGP SIGNATURE-----
--PW0Eas8rCkcu1VkF--
3145
days inactive
3207
days old
15 comments
5 participants
participants (5)
-
Bill James -
Niels de Vos -
Nir Soffer -
Ravishankar N -
Roderick Mooi