Re: [ovirt-users] ovirt glusterfs performance

Friday, 12 February 2016

--------------080509040202050109010204
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 8bit

wow, that made a whole lot of difference!
Thank you!

[root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile1 bs=1M 
count=1000 oflag=direct
1048576000 bytes (1.0 GB) copied, 20.2778 s, 51.7 MB/s

these are the options now for the record.

Options Reconfigured:
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
performance.write-behind: off
performance.write-behind-window-size: 4MB
performance.cache-size: 1GB
performance.readdir-ahead: on

Thanks again!

On 2/11/16 8:18 PM, Ravishankar N wrote:
...
 Hi Bill,
 Can you enable virt-profile setting for your volume and see if that 
 helps? You need to enable this optimization when you create the volume 
 using ovrit, or use the following command for an existing volume:

 #gluster volume set <volname> group virt

 -Ravi

 On 02/12/2016 05:22 AM, Bill James wrote:
> My apologies, I'm showing how much of a noob I am.
> Ignore last direct to gluster numbers, as that wasn't really glusterfs.
>
>
> [root@ovirt2 test ~]# mount -t glusterfs 
> ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/
> [root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2 
> bs=1M count=1000 oflag=direct
> 1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s
>
> That's more how I expected, it is pointing to glusterfs performance.
>
>
>
> On 02/11/2016 03:27 PM, Bill James wrote:
>> don't know if it helps, but I ran a few more tests, all from the 
>> same hardware node.
>>
>> The VM:
>> [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile bs=1M 
>> count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s
>>
>> Writing directly to gluster volume:
>> [root@ovirt2 test ~]# time dd if=/dev/zero 
>> of=/gluster-store/brick1/gv1/testfile bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s
>>
>>
>> Writing to NFS volume:
>> [root@ovirt2 test ~]# time dd if=/dev/zero 
>> of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s
>>
>> NFS & Gluster are using the same interface. Tests were not run at 
>> same time.
>>
>> This would suggest my problem isn't glusterfs, but the VM performance.
>>
>>
>>
>> On 02/11/2016 03:13 PM, Bill James wrote:
>>> xml attached.
>>>
>>>
>>> On 02/11/2016 12:28 PM, Nir Soffer wrote:
>>>> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james(a)j2.com&gt; 
>>>> wrote:
>>>>> thank you for the reply.
>>>>>
>>>>> We setup gluster using the names associated with  NIC 2 IP.
>>>>>   Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>   Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>   Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>
>>>>> That's NIC 2's IP.
>>>>> Using 'iftop -i eno2 -L 5 -t' :
>>>>>
>>>>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>>>>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
>>>> Can you share the xml of this vm? You can find it in vdsm log,
>>>> at the time you start the vm.
>>>>
>>>> Or you can do (on the host):
>>>>
>>>> # virsh
>>>> virsh # list
>>>> (username: vdsm@ovirt password: shibboleth)
>>>> virsh # dumpxml vm-id
>>>>
>>>>> Peak rate (sent/received/total):                      281Mb 5.36Mb
>>>>> 282Mb
>>>>> Cumulative (sent/received/total): 1.96GB 14.6MB
>>>>> 1.97GB
>>>>>
>>>>> gluster volume info gv1:
>>>>>   Options Reconfigured:
>>>>>   performance.write-behind-window-size: 4MB
>>>>>   performance.readdir-ahead: on
>>>>>   performance.cache-size: 1GB
>>>>>   performance.write-behind: off
>>>>>
>>>>> performance.write-behind: off didn't help.
>>>>> Neither did any other changes I've tried.
>>>>>
>>>>>
>>>>> There is no VM traffic on this VM right now except my test.
>>>>>
>>>>>
>>>>>
>>>>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>>>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N 
>>>>>> <ravishankar(a)redhat.com&gt;
>>>>>> wrote:
>>>>>>> +gluster-users
>>>>>>>
>>>>>>> Does disabling 'performance.write-behind' give a
better 
>>>>>>> throughput?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>>>>> I'm setting up a ovirt cluster using glusterfs and
noticing 
>>>>>>>> not stellar
>>>>>>>> performance.
>>>>>>>> Maybe my setup could use some adjustments?
>>>>>>>>
>>>>>>>> 3 hardware nodes running centos7.2, glusterfs 3.7.6.1,
ovirt 
>>>>>>>> 3.6.2.6-1.
>>>>>>>> Each node has 8 spindles configured in 1 array which is
split 
>>>>>>>> using LVM
>>>>>>>> with one logical volume for system and one for gluster.
>>>>>>>> They each have 4 NICs,
>>>>>>>>    NIC1 = ovirtmgmt
>>>>>>>>    NIC2 = gluster  (1GbE)
>>>>>> How do you ensure that gluster trafic is using this nic?
>>>>>>
>>>>>>>>    NIC3 = VM traffic
>>>>>> How do you ensure that vm trafic is using this nic?
>>>>>>
>>>>>>>> I tried with default glusterfs settings
>>>>>> And did you find any difference?
>>>>>>
>>>>>>>> and also with:
>>>>>>>> performance.cache-size: 1GB
>>>>>>>> performance.readdir-ahead: on
>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>
>>>>>>>> [root@ovirt3 test scripts]# gluster volume info gv1
>>>>>>>>
>>>>>>>> Volume Name: gv1
>>>>>>>> Type: Replicate
>>>>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>>>>> Status: Started
>>>>>>>> Number of Bricks: 1 x 3 = 3
>>>>>>>> Transport-type: tcp
>>>>>>>> Bricks:
>>>>>>>> Brick1:
ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>>> Brick2:
ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>>> Brick3:
ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>>>>> Options Reconfigured:
>>>>>>>> performance.cache-size: 1GB
>>>>>>>> performance.readdir-ahead: on
>>>>>>>> performance.write-behind-window-size: 4MB
>>>>>>>>
>>>>>>>>
>>>>>>>> Using simple dd test on VM in ovirt:
>>>>>>>>     dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>> block size of 1G?!
>>>>>>
>>>>>> Try 1M (our default for storage operations)
>>>>>>
>>>>>>>>         1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3
MB/s
>>>>>>>>
>>>>>>>> Another VM not in ovirt using nfs:
>>>>>>>>      dd if=/dev/zero of=/root/testfile bs=1G count=1
oflag=direct
>>>>>>>>         1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8
MB/s
>>>>>>>>
>>>>>>>>
>>>>>>>> Is that expected or is there a better way to set it up to
get 
>>>>>>>> better
>>>>>>>> performance?
>>>>>> Adding Niels for advice.
>>>>>>
>>>>>>>> This email, its contents and ....
>>>>>> Please avoid this, this is a public mailing list, everything you

>>>>>> write
>>>>>> here is public.
>>>>>>
>>>>>> Nir
>>>>> I'll have to look into how to remove this sig for this mailing 
>>>>> list....
>>>>>
>>>>> Cloud Services for Business www.j2.com
>>>>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>>>>>
>>>>>
>>>>> This email, its contents and attachments contain information from 
>>>>> j2 Global,
>>>>> Inc. and/or its affiliates which may be privileged, confidential or
>>>>> otherwise protected from disclosure. The information is intended 
>>>>> to be for
>>>>> the addressee(s) only. If you are not an addressee, any 
>>>>> disclosure, copy,
>>>>> distribution, or use of the contents of this message is 
>>>>> prohibited. If you
>>>>> have received this email in error please notify the sender by 
>>>>> reply e-mail
>>>>> and delete the original message and any copies. (c) 2015 j2 
>>>>> Global, Inc. All
>>>>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, 
>>>>> and Onebox
>>>>> are registered trademarks of j2 Global, Inc. and its affiliates.
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>> www.j2.com 
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>>
>> This email, its contents and attachments contain information from j2 
>> Global, Inc 
>>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.

>> and/or its affiliates which may be privileged, confidential or 
>> otherwise protected from disclosure. The information is intended to 
>> be for the addressee(s) only. If you are not an addressee, any 
>> disclosure, copy, distribution, or use of the contents of this 
>> message is prohibited. If you have received this email in error 
>> please notify the sender by reply e-mail and delete the original 
>> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;. 
>> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ® 
>> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;,

>> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ® 
>> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/> 
>> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. 
>> and its affiliates.
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> www.j2.com 
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...
>
> This email, its contents and attachments contain information from j2 
> Global, Inc 
>
<http://www.j2.com/?utm_source=j2global&utm_medium=xsell-referral&...;.

> and/or its affiliates which may be privileged, confidential or 
> otherwise protected from disclosure. The information is intended to 
> be for the addressee(s) only. If you are not an addressee, any 
> disclosure, copy, distribution, or use of the contents of this 
> message is prohibited. If you have received this email in error 
> please notify the sender by reply e-mail and delete the original 
> message and any copies. © 2015 j2 Global, Inc <http://www.j2.com/>;. 
> All rights reserved. eFax ® <http://www.efax.com/>;, eVoice ® 
> <http://www.evoice.com/>;, Campaigner ® <http://www.campaigner.com/>;, 
> FuseMail ® <http://www.fusemail.com/>;, KeepItSafe ® 
> <http://www.keepitsafe.com/> and Onebox ® <http://www.onebox.com/> 
> are r egistered trademarks of j2 Global, Inc <http://www.j2.com/>;. 
> and its affiliates.
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

Cloud Services for Business www.j2.com
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox

This email, its contents and attachments contain information from j2 Global, Inc. and/or
its affiliates which may be privileged, confidential or otherwise protected from
disclosure. The information is intended to be for the addressee(s) only. If you are not an
addressee, any disclosure, copy, distribution, or use of the contents of this message is
prohibited. If you have received this email in error please notify the sender by reply
e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights
reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered
trademarks of j2 Global, Inc. and its affiliates.

--------------080509040202050109010204
Content-Type: text/html; charset="windows-1252"
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    wow, that made a whole lot of difference!<br>
    Thank you!<br>
    <br>
    [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile1 bs=1M
    count=1000 oflag=direct<br>
    1048576000 bytes (1.0 GB) copied, 20.2778 s, 51.7 MB/s<br>
    <br>
    these are the options now for the record.<br>
    <br>
    Options Reconfigured:<br>
    cluster.server-quorum-type: server<br>
    cluster.quorum-type: auto<br>
    network.remote-dio: enable<br>
    cluster.eager-lock: enable<br>
    performance.stat-prefetch: off<br>
    performance.io-cache: off<br>
    performance.read-ahead: off<br>
    performance.quick-read: off<br>
    performance.write-behind: off<br>
    performance.write-behind-window-size: 4MB<br>
    performance.cache-size: 1GB<br>
    performance.readdir-ahead: on<br>
    <br>
    <br>
    Thanks again!<br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 2/11/16 8:18 PM, Ravishankar N
      wrote:<br>
    </div>
    <blockquote cite="mid:56BD5D15.6060607@redhat.com"
type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <div class="moz-cite-prefix">Hi Bill,<br>
        Can you enable virt-profile setting for your volume and see if
        that helps? You need to enable this optimization when you create
        the volume using ovrit, or use the following command for an
        existing volume:<br>
        <tt><br>
          #gluster volume set &lt;volname&gt; group virt</tt><br>
        <br>
        -Ravi<br>
        <br>
        <br>
        On 02/12/2016 05:22 AM, Bill James wrote:<br>
      </div>
      <blockquote cite="mid:56BD1EB9.5000802@j2.com"
type="cite"> My
        apologies, I'm showing how much of a noob I am.<br>
        Ignore last direct to gluster numbers, as that wasn't really
        glusterfs.<br>
        <br>
        <br>
        [root@ovirt2 test ~]# mount -t glusterfs
        ovirt2-ks.test.j2noc.com:/gv1 /mnt/tmp/<br>
        [root@ovirt2 test ~]# time dd if=/dev/zero of=/mnt/tmp/testfile2
        bs=1M count=1000 oflag=direct<br>
        1048576000 bytes (1.0 GB) copied, 65.8596 s, 15.9 MB/s<br>
        <br>
        That's more how I expected, it is pointing to glusterfs
        performance.<br>
        <br>
        <br>
        <br>
        <div class="moz-cite-prefix">On 02/11/2016 03:27 PM, Bill James
          wrote:<br>
        </div>
        <blockquote cite="mid:56BD18D7.8080909@j2.com"
type="cite">
          don't know if it helps, but I ran a few more tests, all from
          the same hardware node.<br>
          <br>
          The VM:<br>
          [root@billjov1 ~]# time dd if=/dev/zero of=/root/testfile
          bs=1M count=1000 oflag=direct<br>
          1048576000 bytes (1.0 GB) copied, 62.5535 s, 16.8 MB/s<br>
          <br>
          Writing directly to gluster volume:<br>
          [root@ovirt2 test ~]# time dd if=/dev/zero
          of=/gluster-store/brick1/gv1/testfile bs=1M count=1000
          oflag=direct<br>
          1048576000 bytes (1.0 GB) copied, 9.92048 s, 106 MB/s<br>
          <br>
          <br>
          Writing to NFS volume:<br>
          [root@ovirt2 test ~]# time dd if=/dev/zero
          of=/mnt/storage/qa/testfile bs=1M count=1000 oflag=direct<br>
          1048576000 bytes (1.0 GB) copied, 10.5776 s, 99.1 MB/s<br>
          <br>
          NFS &amp; Gluster are using the same interface. Tests were not
          run at same time.<br>
          <br>
          This would suggest my problem isn't glusterfs, but the VM
          performance.<br>
          <br>
          <br>
          <br>
          <div class="moz-cite-prefix">On 02/11/2016 03:13 PM, Bill
            James wrote:<br>
          </div>
          <blockquote cite="mid:56BD1589.8090506@j2.com"
type="cite">xml
            attached. <br>
            <br>
            <br>
            On 02/11/2016 12:28 PM, Nir Soffer wrote: <br>
            <blockquote type="cite">On Thu, Feb 11, 2016 at 8:27 PM,
              Bill James <a moz-do-not-send="true"
                class="moz-txt-link-rfc2396E"

href="mailto:bill.james@j2.com">&lt;bill.james@j2.com&gt;</a>
              wrote: <br>
              <blockquote type="cite">thank you for the reply. <br>
                <br>
                We setup gluster using the names associated with  NIC 2
                IP. <br>
                  Brick1:
                ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
                  Brick2:
                ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
                  Brick3:
                ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1 <br>
                <br>
                That's NIC 2's IP. <br>
                Using 'iftop -i eno2 -L 5 -t' : <br>
                <br>
                dd if=/dev/zero of=/root/testfile bs=1M count=1000
                oflag=direct <br>
                1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s <br>
              </blockquote>
              Can you share the xml of this vm? You can find it in vdsm
              log, <br>
              at the time you start the vm. <br>
              <br>
              Or you can do (on the host): <br>
              <br>
              # virsh <br>
              virsh # list <br>
              (username: vdsm@ovirt password: shibboleth) <br>
              virsh # dumpxml vm-id <br>
              <br>
              <blockquote type="cite">Peak rate
                (sent/received/total):                      281Mb 5.36Mb
                <br>
                282Mb <br>
                Cumulative (sent/received/total):                   
                1.96GB 14.6MB <br>
                1.97GB <br>
                <br>
                gluster volume info gv1: <br>
                  Options Reconfigured: <br>
                  performance.write-behind-window-size: 4MB <br>
                  performance.readdir-ahead: on <br>
                  performance.cache-size: 1GB <br>
                  performance.write-behind: off <br>
                <br>
                performance.write-behind: off didn't help. <br>
                Neither did any other changes I've tried. <br>
                <br>
                <br>
                There is no VM traffic on this VM right now except my
                test. <br>
                <br>
                <br>
                <br>
                On 02/10/2016 11:55 PM, Nir Soffer wrote: <br>
                <blockquote type="cite">On Thu, Feb 11, 2016 at 2:42 AM,
                  Ravishankar N <a moz-do-not-send="true"
                    class="moz-txt-link-rfc2396E"

href="mailto:ravishankar@redhat.com">&lt;ravishankar@redhat.com&gt;</a>
                  <br>
                  wrote: <br>
                  <blockquote type="cite">+gluster-users <br>
                    <br>
                    Does disabling 'performance.write-behind' give a
                    better throughput? <br>
                    <br>
                    <br>
                    <br>
                    On 02/10/2016 11:06 PM, Bill James wrote: <br>
                    <blockquote type="cite">I'm setting up a ovirt
                      cluster using glusterfs and noticing not stellar <br>
                      performance. <br>
                      Maybe my setup could use some adjustments? <br>
                      <br>
                      3 hardware nodes running centos7.2, glusterfs
                      3.7.6.1, ovirt 3.6.2.6-1. <br>
                      Each node has 8 spindles configured in 1 array
                      which is split using LVM <br>
                      with one logical volume for system and one for
                      gluster. <br>
                      They each have 4 NICs, <br>
                         NIC1 = ovirtmgmt <br>
                         NIC2 = gluster  (1GbE) <br>
                    </blockquote>
                  </blockquote>
                  How do you ensure that gluster trafic is using this
                  nic? <br>
                  <br>
                  <blockquote type="cite">
                    <blockquote type="cite">   NIC3 = VM traffic
<br>
                    </blockquote>
                  </blockquote>
                  How do you ensure that vm trafic is using this nic? <br>
                  <br>
                  <blockquote type="cite">
                    <blockquote type="cite">I tried with default
                      glusterfs settings <br>
                    </blockquote>
                  </blockquote>
                  And did you find any difference? <br>
                  <br>
                  <blockquote type="cite">
                    <blockquote type="cite">and also with: <br>
                      performance.cache-size: 1GB <br>
                      performance.readdir-ahead: on <br>
                      performance.write-behind-window-size: 4MB <br>
                      <br>
                      [root@ovirt3 test scripts]# gluster volume info
                      gv1 <br>
                      <br>
                      Volume Name: gv1 <br>
                      Type: Replicate <br>
                      Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2 <br>
                      Status: Started <br>
                      Number of Bricks: 1 x 3 = 3 <br>
                      Transport-type: tcp <br>
                      Bricks: <br>
                      Brick1:
                      ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
                      <br>
                      Brick2:
                      ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
                      <br>
                      Brick3:
                      ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
                      <br>
                      Options Reconfigured: <br>
                      performance.cache-size: 1GB <br>
                      performance.readdir-ahead: on <br>
                      performance.write-behind-window-size: 4MB <br>
                      <br>
                      <br>
                      Using simple dd test on VM in ovirt: <br>
                          dd if=/dev/zero of=/root/testfile bs=1G
                      count=1 oflag=direct <br>
                    </blockquote>
                  </blockquote>
                  block size of 1G?! <br>
                  <br>
                  Try 1M (our default for storage operations) <br>
                  <br>
                  <blockquote type="cite">
                    <blockquote type="cite">        1073741824 bytes
                      (1.1 GB) copied, 65.9337 s, 16.3 MB/s <br>
                      <br>
                      Another VM not in ovirt using nfs: <br>
                           dd if=/dev/zero of=/root/testfile bs=1G
                      count=1 oflag=direct <br>
                              1073741824 bytes (1.1 GB) copied, 27.0079
                      s, 39.8 MB/s <br>
                      <br>
                      <br>
                      Is that expected or is there a better way to set
                      it up to get better <br>
                      performance? <br>
                    </blockquote>
                  </blockquote>
                  Adding Niels for advice. <br>
                  <br>
                  <blockquote type="cite">
                    <blockquote type="cite">This email, its contents and
                      .... <br>
                    </blockquote>
                  </blockquote>
                  Please avoid this, this is a public mailing list,
                  everything you write <br>
                  here is public. <br>
                  <br>
                  Nir <br>
                </blockquote>
                I'll have to look into how to remove this sig for this
                mailing list.... <br>
                <br>
                Cloud Services for Business <a moz-do-not-send="true"
                  class="moz-txt-link-abbreviated"
                  href="http://www.j2.com">www.j2.com</a> <br>
                j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe
                | Onebox <br>
                <br>
                <br>
                This email, its contents and attachments contain
                information from j2 Global, <br>
                Inc. and/or its affiliates which may be privileged,
                confidential or <br>
                otherwise protected from disclosure. The information is
                intended to be for <br>
                the addressee(s) only. If you are not an addressee, any
                disclosure, copy, <br>
                distribution, or use of the contents of this message is
                prohibited. If you <br>
                have received this email in error please notify the
                sender by reply e-mail <br>
                and delete the original message and any copies. (c) 2015
                j2 Global, Inc. All <br>
                rights reserved. eFax, eVoice, Campaigner, FuseMail,
                KeepItSafe, and Onebox <br>
                are registered trademarks of j2 Global, Inc. and its
                affiliates. <br>
              </blockquote>
            </blockquote>
            <br>
            <br>
            <fieldset class="mimeAttachmentHeader"></fieldset>
            <br>
            <pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
          </blockquote>
          <br>
          <p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&amp;utm_medium=xsel...
                style="color:windowtext; text-decoration:none"><img
                  moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
                  alt="www.j2.com" height="46" border="0"
width="391"></span></a></p>
          <p><span
style="font-size:8.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:gray">This

              email, its contents and attachments contain information
              from <a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&amp;utm_medium=xsel...

                Global, Inc</a>. and/or its affiliates which may be
              privileged, confidential or otherwise protected from
              disclosure. The information is intended to be for the
              addressee(s) only. If you are not an addressee, any
              disclosure, copy, distribution, or use of the contents of
              this message is prohibited. If you have received this
              email in error please notify the sender by reply e-mail
              and delete the original message and any copies. © 2015 <a
                moz-do-not-send="true"
href="http://www.j2.com/">j2
                Global, Inc</a>. All rights reserved. <a
                moz-do-not-send="true"
href="http://www.efax.com/">eFax
                ®</a>, <a moz-do-not-send="true"
                href="http://www.evoice.com/">eVoice ®</a>, <a
                moz-do-not-send="true"
href="http://www.campaigner.com/">Campaigner
                ®</a>, <a moz-do-not-send="true"
                href="http://www.fusemail.com/">FuseMail ®</a>, <a
                moz-do-not-send="true"
href="http://www.keepitsafe.com/">KeepItSafe
                ®</a> and <a moz-do-not-send="true"
                href="http://www.onebox.com/">Onebox ®</a> are r
              egistered trademarks of <a moz-do-not-send="true"
                href="http://www.j2.com/">j2 Global, Inc</a>. and its
              affiliates.</span></p>
          <br>
          <fieldset class="mimeAttachmentHeader"></fieldset>
          <br>
          <pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
        </blockquote>
        <br>
        <p><a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&amp;utm_medium=xsel...
              style="color:windowtext; text-decoration:none"><img
                moz-do-not-send="true"
src="http://home.j2.com/j2_Global_Cloud_Services/j2_Global_Email_Foo...
                alt="www.j2.com" height="46" border="0"
width="391"></span></a></p>
        <p><span
style="font-size:8.0pt;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:gray">This

            email, its contents and attachments contain information from
            <a moz-do-not-send="true"
href="http://www.j2.com/?utm_source=j2global&amp;utm_medium=xsel...

              Global, Inc</a>. and/or its affiliates which may be
            privileged, confidential or otherwise protected from
            disclosure. The information is intended to be for the
            addressee(s) only. If you are not an addressee, any
            disclosure, copy, distribution, or use of the contents of
            this message is prohibited. If you have received this email
            in error please notify the sender by reply e-mail and delete
            the original message and any copies. © 2015 <a
              moz-do-not-send="true" href="http://www.j2.com/">j2
              Global, Inc</a>. All rights reserved. <a
              moz-do-not-send="true"
href="http://www.efax.com/">eFax ®</a>,
            <a moz-do-not-send="true"
href="http://www.evoice.com/">eVoice

              ®</a>, <a moz-do-not-send="true"
              href="http://www.campaigner.com/">Campaigner ®</a>,
<a
              moz-do-not-send="true"
href="http://www.fusemail.com/">FuseMail

              ®</a>, <a moz-do-not-send="true"
              href="http://www.keepitsafe.com/">KeepItSafe ®</a> and
<a
              moz-do-not-send="true"
href="http://www.onebox.com/">Onebox

              ®</a> are r egistered trademarks of <a
              moz-do-not-send="true" href="http://www.j2.com/">j2
              Global, Inc</a>. and its affiliates.</span></p>
        <br>
        <fieldset class="mimeAttachmentHeader"></fieldset>
        <br>
        <pre wrap="">_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
      </blockquote>
      <br>
      <br>
    </blockquote>
    <br>

This email, its contents and attachments contain information from <a
href="http://www.j2.com/?utm_source=j2global&utm_medium=xsell-re...
Global, Inc</a>. and/or its affiliates which may be privileged, confidential or
otherwise protected from disclosure. The information is intended to be for the
addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use
of the contents of this message is prohibited. If you have received this email in error
please notify the sender by reply e-mail and delete the original message and any copies. ©
2015 <a href="http://www.j2.com/">j2 Global, Inc</a>. All rights
reserved. <a href="http://www.efax.com/">eFax ®</a>, <a
href="http://www.evoice.com/">eVoice ®</a>, <a
href="http://www.campaigner.com/">Campaigner ®</a>, <a
href="http://www.fusemail.com/">FuseMail ®</a>, <a
href="http://www.keepitsafe.com/">KeepItSafe ®</a> and <a
href="http://www.onebox.com/">Onebox ®</a> are registered trademarks of
<a href="http://www.j2.com/">j2 Global, Inc</a>. and its
affiliates.</body>
</html>

--------------080509040202050109010204--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] ovirt glusterfs performance