8:41 a.m.
Underlying FS is XFS
GlusterFS - glusterfs-3.5.0-2.el6
I'm using CentOS, if this is problem could RHEL
packages be used? why CentOS packages are compiled
without native glusterfs support?
Citējot Paul Robert Marino <prmarino1(a)gmail.com> :
What's the underlying filesystem for gluster is
it XFS?
What version of gluster are you using?
What distro are you using and if its not RHEL
or Fedora are you using a version of
QEMU with
gluster support compiled in keep in mind the
versions with CentOS and Scientific Linux do not
include Gluster native support compiled in.
-- Sent from my HP Pre3
On May 11, 2014 5:40, Vadims Korsaks
<truckk(a)inbox.lv> wrote:
Citējot Vijay Bellur <vbellur(a)redhat.com> :
> On 05/11/2014 02:04 AM, Vadims Korsaks wrote:
> > HI!
> >
> > Created 2 node setup with oVirt 3.4 and
CentOS 6.5, for storage created
> > 2 node replicated gluster (3.5) fs on same
hosts with oVirt.
> > mount looks like this:
> > 127.0.0.1:/gluster01 on
> >
/rhev/data-center/mnt/glusterSD/127.0.0.1:_gluster01
type fuse.glusterfs
> >
(rw,default_permissions,allow_other,max_read=131072)
> >
> > when i making gluster test with dd, something
like
> > dd if=/dev/zero bs=1M count=20000
> >
of=/rhev/data-center/mnt/glusterSD/127.0.0.1\:_gluster01/kaka
> > i'm gettting speed ~ 110 MB/s, so this is
1Gbps speed of ethernet adapter
> >
> > but with in VM created in oVirt speed is
lower than 20 MB/s
> >
> > why there is so huge difference?
> > how can improve VMs disks speed?
> >
>
> What are your gluster volume settings? Have you
applied the following
> performance tunables in gluster's virt profile:
>
> eager-lock=enable
> remote-dio=enable
>
> Regards,
> Vijay
>
setting were:
[root@centos155 ~]# gluster volume info gluster01
Volume Name: gluster01
Type: Replicate
Volume ID: 436edaa3-ac8b-421f-aa35-68b5bd7064b6
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.2.75.152:/mnt/gluster01/brick
Brick2: 10.2.75.155:/mnt/gluster01/brick
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
add your settings settings now it looks
[root@centos155 ~]# gluster volume info gluster01
Volume Name: gluster01
Type: Replicate
Volume ID: 436edaa3-ac8b-421f-aa35-68b5bd7064b6
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.2.75.152:/mnt/gluster01/brick
Brick2: 10.2.75.155:/mnt/gluster01/brick
Options Reconfigured:
network.remote-dio: enable
cluster.eager-lock: enable
storage.owner-gid: 36
storage.owner-uid: 36
but this didn't affected performace in any big way
should hosts to be restarted?
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3:58 p.m.
New subject: gluster performance oVirt 3.4
I've got the same issue using the same versions as Korsaks, using CentOS
6.5, too.
I could launch vms with gluster block backend driver using the standard
package of libvirt from centos repo.
but ovirt does not run the VM with the qemu GFAPI integration, the path of
the disk using the fuse mount.
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none'
error_policy='stop'
io='threads'/>
<source
file='/rhev/data-center/mnt/glusterSD/localhost:VMDATA/0add493f-0a7f-4b32-bcd5-ff25ca504b8b/images/68dbbc67-ea24-45a9-8727-4f85d100d1bb/8fe386f7-2aeb-43c2-bcb0-f76829c876b4'>
<seclabel model='selinux' relabel='no'/>
</source>
<target dev='vda' bus='virtio'/>
<serial></serial>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
</disk>
or am i wrong?
On Mon, May 12, 2014 at 7:41 AM, Vadims Korsaks <truckk(a)inbox.lv> wrote:
Underlying FS is XFS
GlusterFS - glusterfs-3.5.0-2.el6
I'm using CentOS, if this is problem could RHEL
packages be used? why CentOS packages are compiled
without native glusterfs support?
Citējot Paul Robert Marino <prmarino1(a)gmail.com> :
> What's the underlying filesystem for gluster is
it XFS?
> What version of gluster are you using?
> What distro are you using and if its not RHEL
or Fedora are you using a version of QEMU with
gluster support compiled in keep in mind the
versions with CentOS and Scientific Linux do not
include Gluster native support compiled in.
>
>
>
> -- Sent from my HP Pre3
>
> On May 11, 2014 5:40, Vadims Korsaks
<truckk(a)inbox.lv> wrote:
>
> Citējot Vijay Bellur <vbellur(a)redhat.com> :
> > On 05/11/2014 02:04 AM, Vadims Korsaks wrote:
> > > HI!
> > >
> > > Created 2 node setup with oVirt 3.4 and
> CentOS 6.5, for storage created
> > > 2 node replicated gluster (3.5) fs on same
> hosts with oVirt.
> > > mount looks like this:
> > > 127.0.0.1:/gluster01 on
> > >
>
/rhev/data-center/mnt/glusterSD/127.0.0.1:_gluster01
> type fuse.glusterfs
> > >
>
(rw,default_permissions,allow_other,max_read=131072)
> > >
> > > when i making gluster test with dd, something
> like
> > > dd if=/dev/zero bs=1M count=20000
> > >
>
of=/rhev/data-center/mnt/glusterSD/127.0.0.1\:_gluster01/kaka
> > > i'm gettting speed ~ 110 MB/s, so this is
> 1Gbps speed of ethernet adapter
> > >
> > > but with in VM created in oVirt speed is
> lower than 20 MB/s
> > >
> > > why there is so huge difference?
> > > how can improve VMs disks speed?
> > >
> >
> > What are your gluster volume settings? Have you
> applied the following
> > performance tunables in gluster's virt profile:
> >
> > eager-lock=enable
> > remote-dio=enable
> >
> > Regards,
> > Vijay
> >
> setting were:
> [root@centos155 ~]# gluster volume info gluster01
>
> Volume Name: gluster01
> Type: Replicate
> Volume ID: 436edaa3-ac8b-421f-aa35-68b5bd7064b6
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.2.75.152:/mnt/gluster01/brick
> Brick2: 10.2.75.155:/mnt/gluster01/brick
> Options Reconfigured:
> storage.owner-gid: 36
> storage.owner-uid: 36
>
>
> add your settings settings now it looks
>
> [root@centos155 ~]# gluster volume info gluster01
>
> Volume Name: gluster01
> Type: Replicate
> Volume ID: 436edaa3-ac8b-421f-aa35-68b5bd7064b6
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.2.75.152:/mnt/gluster01/brick
> Brick2: 10.2.75.155:/mnt/gluster01/brick
> Options Reconfigured:
> network.remote-dio: enable
> cluster.eager-lock: enable
> storage.owner-gid: 36
> storage.owner-uid: 36
>
>
> but this didn't affected performace in any big way
> should hosts to be restarted?
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:09 p.m.
New subject: gluster performance oVirt 3.4
On Mon, May 12, 2014 at 2:58 PM, Tobias Honacker <tobias(a)honacker.info>wrote:
I've got the same issue using the same versions as Korsaks, using
CentOS
6.5, too.
I could launch vms with gluster block backend driver using the standard
package of libvirt from centos repo.
but ovirt does not run the VM with the qemu GFAPI integration, the path
of the disk using the fuse mount.
[snip]
or am i wrong?
This same question asked many times latest months.
If you search through archives we remained here:
http://lists.ovirt.org/pipermail/users/2014-April/023825.html
BTW, Itamar, are there any bug trackers for this still remaining problems
with live migration?
Could we put anything on the wiki? If so, where, so that I can fill this up?
Gianluca
5:05 p.m.
New subject: gluster performance oVirt 3.4
On 05/12/2014 09:09 AM, Gianluca Cecchi wrote:
On Mon, May 12, 2014 at 2:58 PM, Tobias Honacker
<tobias(a)honacker.info
<mailto:tobias@honacker.info>> wrote:
I've got the same issue using the same versions as Korsaks, using
CentOS 6.5, too.
I could launch vms with gluster block backend driver using the
standard package of libvirt from centos repo.
but ovirt does not run the VM with the qemu GFAPI integration, the
path of the disk using the fuse mount.
[snip]
or am i wrong?
This same question asked many times latest months.
If you search through archives we remained here:
http://lists.ovirt.org/pipermail/users/2014-April/023825.html
BTW, Itamar, are there any bug trackers for this still remaining
problems with live migration?
Could we put anything on the wiki? If so, where, so that I can fill this up?
I'd put it under ovirt.org/Troubleshooting (I think we have such a
page), pointing to "Troubleshooting_LiveMigration" page
9:38 p.m.
New subject: gluster performance oVirt 3.4
--Alternative_=_Boundary_=_1399919906
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On the Gluster site there is a QEMU repo I'll send you the link latter toda=
y. But essentially CentOS doesn't include the Gluster client or libraries s=
o they can't compile against it.<br>I'm not sure why CentOS and Scientific
=
Linux don't include it but I think it might just be an over site since Glus=
ter 3.4 is new set of packages which were added in the latest release of RH=
EL 6.<br>In truth you can simply unpack the source RPM and rebuild it with =
Gluster support its not difficult but I don't remember the flags you need t=
o pass the rpmbuild command off the top of my head. That said I'd probably =
use the ones off the Gluster site any way. <br>If you are using
G=
luster 3.5 the ones included in RHEL are incompatible because Gluster still=
has large API changes between minor releases and they were compiled agains=
t 3.4. <br><br>Also Gluster 3.5 is a brand new release so I
wouldn't r=
ule out the possibility of a bug. So if the Gluster enabled QEMU =
rpms don't help you may need to inquire on the Gluster mailing list.<br><br=
<div><font color=3D"#333333" face=3D"arial,
sans-serif"><span style=3D"line=
-height:17px">but ovirt does not run the VM with the
qemu=C2=A0</span></fon=
t><span style=3D"color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;fo=
nt-size:12px;line-height:18px">GFAPI integration, the path of the disk usin=
g the fuse mount.</span></div
<div><font color=3D"#000000" face=3D"Helvetica, Arial,
sans-serif"><span st=
yle=3D"font-size:12px;line-height:18px"><br></span></font></div><div><font
=
color=3D"#000000" face=3D"Helvetica, Arial, sans-serif"><span
style=3D"font=
-size:12px;line-height:18px"><div
<disk type=3D'file' device=3D'disk'
snapshot=3D'no&#=
39;></div><div>=C2=A0 =C2=A0 =C2=A0 <driver
name=3D'qemu' typ=
e=3D'raw' cache=3D'none'
error_policy=3D'stop' io=
=3D'threads'/></div
<div>=C2=A0 =C2=A0 =C2=A0 <source
file=3D'/rhev/data-center/mnt/glus=
terSD/localhost:VMDATA/0add493f-0a7f-4b32-bcd5-ff25ca504b8b/images/68dbbc67=
-ea24-45a9-8727-4f85d100d1bb/8fe386f7-2aeb-43c2-bcb0-f76829c876b4'><=
/div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 <seclabel
model=3D'selinux'=
; relabel=3D'no'/></div
<div>=C2=A0 =C2=A0 =C2=A0 </source></div><div>=C2=A0
=C2=A0 =C2=A0 &l=
t;target dev=3D'vda'
bus=3D'virtio'/></div><div>=C2=A0 =
=C2=A0 =C2=A0 <serial></serial></div><div>=C2=A0
=C2=A0 =C2=A0 =
<alias name=3D'virtio-disk0'/></div><div>=C2=A0
=C2=A0 =C2=A0=
<address type=3D'pci' domain=3D'0x0000'
bus=3D'0x00=
' slot=3D'0x06'
function=3D'0x0'/></div
<div>=C2=A0 =C2=A0
</disk></div></span></font><div><br></div></div><d=
iv><br></div><div>or am i wrong?</div><div
class=3D"gmail_extra"><br><br><d=
iv class=3D"gmail_quote">On Mon, May 12, 2014 at 7:41 AM, Vadims Korsaks
<s=
pan dir=3D"ltr"><<a href=3D"mailto:truckk@inbox.lv"
target=3D"_blank">tr=
uckk(a)inbox.lv</a>&gt;</span> wrote:<br
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Underlying FS is XFS<br GlusterFS - glusterfs-3.5.0-2.el6<br I'm using CentOS, if this is problem could
RHEL<br packages be used? why CentOS
packages are compiled<br without native glusterfs
support?<br <br
Cit=C4=93jot Paul Robert Marino <<a
href=3D"mailto:prmarino1@gmail.com" =
target=3D"_blank">prmarino1(a)gmail.com</a>&gt; :<br <div>> =C2=A0What's the underlying
filesystem for gluster is<br it XFS?<br > =C2=A0What version of gluster are you using?<br > =C2=A0What distro are you using and if its not
RHEL<br or Fedora are you using a version of
QEMU with<br gluster support compiled in keep in
mind the<br versions with CentOS and Scientific
Linux do not<br include Gluster native support
compiled in.<br ><br ><br
><br > =C2=A0-- Sent from my HP
Pre3<br ><br </div><div><div>> =C2=A0 On May 11,
2014 5:40, Vadims Korsaks<br <<a
href=3D"mailto:truckk@inbox.lv"
target=3D"_blank">truckk(a)inbox.lv</a=
> =C2=A0Cit=C4=93jot Vijay Bellur <<a
href=3D"mailto:vbellur@redhat.c=
om" target=3D"_blank">vbellur(a)redhat.com</a>&gt; :<br > =C2=A0> On 05/11/2014 02:04 AM, Vadims Korsaks
wrote:<br > =C2=A0> >
HI!<br > =C2=A0>
><br > =C2=A0> >
Created 2 node setup with oVirt 3.4 and<br >
=C2=A0CentOS 6.5, for storage created<br >
=C2=A0> > 2 node replicated gluster (3.5) fs on same<br > =C2=A0hosts with oVirt.<br > =C2=A0> > mount looks like
this:<br > =C2=A0> >
127.0.0.1:/gluster01 on<br > =C2=A0>
><br ><br /rhev/data-center/mnt/glusterSD/127.0.0.1:_gluster01<br > =C2=A0type fuse.glusterfs<br > =C2=A0> ><br ><br
(rw,default_permissions,allow_other,max_read=3D131072)<br > =C2=A0> ><br > =C2=A0> > when i making gluster test
with dd, something<br > =C2=A0like<br > =C2=A0> > dd if=3D/dev/zero bs=3D1M
count=3D20000<br > =C2=A0>
><br ><br of=3D/rhev/data-center/mnt/glusterSD/<a
href=3D"http://127.0.0.1" target=3D=
"_blank">127.0.0.1</a>\:_gluster01/kaka<br > =C2=A0> > i'm gettting speed ~
110 MB/s, so this is<br > =C2=A01Gbps speed of
ethernet adapter<br > =C2=A0>
><br > =C2=A0> > but
with in VM created in oVirt speed is<br >
=C2=A0lower than 20 MB/s<br > =C2=A0>
><br > =C2=A0> > why
there is so huge difference?<br > =C2=A0> > how
can improve VMs disks speed?<br > =C2=A0>
><br > =C2=A0><br > =C2=A0> What are your gluster volume settings?
Have you<br > =C2=A0applied the
following<br > =C2=A0> performance
tunables in gluster's virt profile:<br >
=C2=A0><br > =C2=A0>
eager-lock=3Denable<br > =C2=A0>
remote-dio=3Denable<br > =C2=A0><br > =C2=A0> Regards,<br > =C2=A0> Vijay<br > =C2=A0><br >
=C2=A0setting were:<br > =C2=A0[root@centos155 ~]#
gluster volume info gluster01<br
><br > =C2=A0Volume Name:
gluster01<br > =C2=A0Type:
Replicate<br > =C2=A0Volume ID:
436edaa3-ac8b-421f-aa35-68b5bd7064b6<br >
=C2=A0Status: Started<br > =C2=A0Number of Bricks: 1 x
2 =3D 2<br > =C2=A0Transport-type:
tcp<br > =C2=A0Bricks:<br > =C2=A0Brick1:
10.2.75.152:/mnt/gluster01/brick<br >
=C2=A0Brick2: 10.2.75.155:/mnt/gluster01/brick<br >
=C2=A0Options Reconfigured:<br > =C2=A0storage.owner-gid:
36<br > =C2=A0storage.owner-uid:
36<br ><br ><br >
=C2=A0add your settings settings now it looks<br
><br > =C2=A0[root@centos155 ~]#
gluster volume info gluster01<br
><br > =C2=A0Volume Name:
gluster01<br > =C2=A0Type:
Replicate<br > =C2=A0Volume ID:
436edaa3-ac8b-421f-aa35-68b5bd7064b6<br >
=C2=A0Status: Started<br > =C2=A0Number of Bricks: 1 x
2 =3D 2<br > =C2=A0Transport-type:
tcp<br > =C2=A0Bricks:<br > =C2=A0Brick1:
10.2.75.152:/mnt/gluster01/brick<br >
=C2=A0Brick2: 10.2.75.155:/mnt/gluster01/brick<br >
=C2=A0Options Reconfigured:<br > =C2=A0network.remote-dio:
enable<br > =C2=A0cluster.eager-lock:
enable<br > =C2=A0storage.owner-gid:
36<br > =C2=A0storage.owner-uid:
36<br ><br ><br >
=C2=A0but this didn't affected performace in any big way<br > =C2=A0should hosts to be restarted?<br ><br >
=C2=A0_______________________________________________<br >
=C2=A0Users mailing list<br > =C2=A0<a
href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt=
=2Eorg</a><br > =C2=A0<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" target=
=3D"_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br <br
_______________________________________________<br Users
mailing list<br <a
href=3D"mailto:Users@ovirt.org"
target=3D"_blank">Users(a)ovirt.org</a><br=
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank"=
>http://lists.ovirt.org/mailman/listinfo/users</a><br
</div></div></blockquote></div><br></div></div
--Alternative_=_Boundary_=_1399919906--
<span style=3D"font-family:Prelude, Verdana,
san-serif;"><br><br></span><s=
pan
id=3D"signature"><div style=3D"font-family: arial, sans-serif;
font-siz=
e: 12px;color: #999999;">-- Sent from my HP
Pre3</div><br></span><span styl=
e=3D"color:navy; font-family:Prelude, Verdana, san-serif; "><hr
align=3D"le=
ft" style=3D"width:75%">On May 12, 2014 8:58, Tobias Honacker
<tobias@ho=
nacker.info> wrote: <br><br></span><div
dir=3D"ltr">I've got the sam=
e issue using the same versions as Korsaks, using CentOS 6.5, too.<div>I co=
uld launch vms with gluster=C2=A0<span style=3D"color:rgb(51,51,51);font-fa=
mily:arial,sans-serif;font-size:13px;line-height:17px">block backend driver=
=C2=A0using the standard package of libvirt from centos repo.</span></div> wrote:<br>
><br
12:33 a.m.
New subject: gluster performance oVirt 3.4
On Mon, May 12, 2014 at 8:38 PM, Paul Robert Marino <prmarino1(a)gmail.com>wrote:
On the Gluster site there is a QEMU repo I'll send you the link
latter
today. But essentially CentOS doesn't include the Gluster client or
libraries so they can't compile against it.
I'm not sure why CentOS and Scientific Linux don't include it but I think
it might just be an over site since Gluster 3.4 is new set of packages
which were added in the latest release of RHEL 6.
In truth you can simply unpack the source RPM and rebuild it with Gluster
support its not difficult but I don't remember the flags you need to pass
the rpmbuild command off the top of my head. That said I'd probably use the
ones off the Gluster site any way.
If you are using Gluster 3.5 the ones included in RHEL are incompatible
because Gluster still has large API changes between minor releases and they
were compiled against 3.4.
Also Gluster 3.5 is a brand new release so I wouldn't rule out the
possibility of a bug. So if the Gluster enabled QEMU rpms don't help you
may need to inquire on the Gluster mailing list.
If I remember correctly, due to problems with snapshots (now solved I
think) and with live migration (not yet solved) the developers hard coded
the fact that the xml domain definition is the suboptimal one.
So, even if you change gluster repo, the qemu command line generated by
oVirt will be the same, so without the use of libgfapi.
Gianluca
4:28 a.m.
New subject: gluster performance oVirt 3.4
----- Original Message -----
From: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com>
To: "Paul Robert Marino" <prmarino1(a)gmail.com>
Cc: "users" <users(a)ovirt.org>
Sent: Monday, May 12, 2014 5:33:41 PM
Subject: Re: [ovirt-users] gluster performance oVirt 3.4
On Mon, May 12, 2014 at 8:38 PM, Paul Robert Marino
<prmarino1(a)gmail.com>wrote:
> On the Gluster site there is a QEMU repo I'll send you the link latter
> today. But essentially CentOS doesn't include the Gluster client or
> libraries so they can't compile against it.
> I'm not sure why CentOS and Scientific Linux don't include it but I think
> it might just be an over site since Gluster 3.4 is new set of packages
> which were added in the latest release of RHEL 6.
> In truth you can simply unpack the source RPM and rebuild it with Gluster
> support its not difficult but I don't remember the flags you need to pass
> the rpmbuild command off the top of my head. That said I'd probably use the
> ones off the Gluster site any way.
> If you are using Gluster 3.5 the ones included in RHEL are incompatible
> because Gluster still has large API changes between minor releases and they
> were compiled against 3.4.
>
> Also Gluster 3.5 is a brand new release so I wouldn't rule out the
> possibility of a bug. So if the Gluster enabled QEMU rpms don't help you
> may need to inquire on the Gluster mailing list.
>
>
If I remember correctly, due to problems with snapshots (now solved I
think) and with live migration (not yet solved) the developers hard coded
the fact that the xml domain definition is the suboptimal one.
So, even if you change gluster repo, the qemu command line generated by
oVirt will be the same, so without the use of libgfapi.
I recently tested with F20 hosts and virt pkgs from the fedora-virt-preview
repo, and the libgfapi support remained disabled.
The changelog for libvirt 1.2.2 mentions a gluster/qemu/snapshot fix, I'd
hoped this would do the trick, but I couldn't figure out how to re-enable
libgfapi to test it...
Jason
Gianluca
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
3661
days inactive
3662
days old
6 comments
6 participants
participants (6)
-
Gianluca Cecchi -
Itamar Heim -
Jason Brooks -
Paul Robert Marino -
Tobias Honacker -
Vadims Korsaks