5:49 p.m.
--_000_DB6P190MB0280A69BAE2A377274B72375C8B40DB6P190MB0280EURP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
--_000_DB6P190MB0280A69BAE2A377274B72375C8B40DB6P190MB0280EURP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p><span>I am willing to assemble a oVirt
"pod", made of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br <span></span></p <p>This said, my first question is: How should I deploy
GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <p>Moacir<br
</p </div </body
</html
--_000_DB6P190MB0280A69BAE2A377274B72375C8B40DB6P190MB0280EURP_--
6:48 a.m.
1) RAID5 may be a performance hit
2) I'd be inclined to do this as JBOD by creating a distributed disperse
volume on each server. Something like
echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e
"server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
3) I think the above
4) Gluster does support tiering, but IIRC you'd need the same number of SSD
as spindle drives. There may be another way to use the SSD as a fast
cache.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.com
wrote:
I am willing to assemble a oVirt "pod", made of 3 servers,
each with 2 CPU
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
a dual 10Gb NIC. So my intention is to create a loop like a server triangle
using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
move VMs around the pod (east /west traffic) while using the 10Gb
interfaces for giving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
then create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:56 p.m.
--_000_VI1P190MB02858CCC4D7DCD6A86090CC9C8B50VI1P190MB0285EURP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi Colin,
I am in Portugal, so sorry for this late response. It is quite confusing fo=
r me, please consider:
1 - What if the RAID is done by the server's disk controller, not by softwa=
re?
2 - For JBOD I am just using gdeploy to deploy it. However, I am not using =
the oVirt node GUI to do this.
3 - As the VM .qcow2 files are quite big, tiering would only help if made b=
y an intelligent system that uses SSD for chunks of data not for the entire=
.qcow2 file. But I guess this is a problem everybody else has. So, Do you =
know how tiering works in Gluster?
4 - I am putting the OS on the first disk. However, would you do differentl=
y?
Moacir
________________________________
From: Colin Coe <colin.coe(a)gmail.com Sent:
Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
1) RAID5 may be a performance hit-
2) I'd be inclined to do this as JBOD by creating a distributed disperse vo=
lume on each server. Something like
echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERV=
ER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
3) I think the above.
4) Gluster does support tiering, but IIRC you'd need the same number of SSD=
as spindle drives. There may be another way to use the SSD as a fast cach=
e.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_VI1P190MB02858CCC4D7DCD6A86090CC9C8B50VI1P190MB0285EURP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Hi Colin,</p
<p><br </p
<p><span>I am in Portugal</span>, so sorry for this late response.
It is qu=
ite confusing for me, please consider:</p
<p></p <div><b><br </b>1<b> - </b>What if the RAID is done by
the server's disk controller, no=
t by software?</div <br
<p></p <p>2 -<b> </b>For
JBOD I am just using gdeploy to deploy it. However, I am =
not using the oVirt node GUI to do this.</p
<p><br </p
<p>3 -<b> </b>As the VM .qcow2 files are quite big,
tiering would only=
help if made by an intelligent system that uses SSD for chunks of data not=
for the entire .qcow2 file. But I guess this is a problem everybody else h=
as. So, Do you know how tiering works
in Gluster?<br </p
<p><br </p
<p>4 - I am putting the OS on the first disk. However, would you do
di=
fferently?<br </p
<p><br </p
Moacir<br <br <div
style=3D"color: rgb(49, 55, 57);" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Colin Coe <colin=
.coe(a)gmail.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 4:48 AM<br <b>To:</b> Moacir
Ferreira<br <b>Cc:</b>
users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
dir=3D"ltr">1) RAID5 may be a performance hit- <br <div><br
</div <div>2) I'd be inclined to
do this as JBOD by creating a distributed disper=
se volume on each server. Something like
<div><br </div <div>echo gluster volume create dispersevol disperse-data
5 redundancy 2 \<=
/div <div>$(for SERVER in a b c; do for BRICK in $(seq 1 5);
do echo -e "se=
rver${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)</d=
iv <div><br
</div <div>3) I think the above.
<b></b></div <div><br </div
<div>4) Gluster does support tiering, but IIRC you'd need the same number
o=
f SSD as spindle drives. There may be another way to use the SSD as a=
fast cache. </div <div><br </div
<div>Where are you putting the OS?</div
<div><br </div <div>Hope I understood the question...</div <div><br
</div <div>Thanks</div </div
</div <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira=
<span dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira(a)hotmail.com</a>&gt;</span> wrote:<br <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_2460985691746498322divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p><span>I am willing to assemble a oVirt
"pod", made of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br <span></span></p <p>This said, my first question is: How should I deploy
GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <span class=3D"HOEnZb"><font
color=3D"#888888" <p>Moacir<br </p
</font></span></div
</div <br
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/<wbr>mailman/...
br <br </blockquote </div
<br </div </div
</div </div </body
</html
--_000_VI1P190MB02858CCC4D7DCD6A86090CC9C8B50VI1P190MB0285EURP_--
2:41 p.m.
Hi
I just thought that you'd do hardware RAID if you had the controller or
JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pretty
likely to have a hardware RAID controller. I've never done JBOD with
hardware RAID. I think having a single gluster brick on hardware JBOD
would be riskier than multiple bricks, each on a single disk, but thats not
based on anything other than my prejudices.
I thought gluster tiering was for the most frequently accessed files, in
which case all the VMs disks would end up in the hot tier. However, I have
been wrong before...
I just wanted to know where the OS was going as I didn't see it mentioned
in the OP. Normally, I'd have the OS on a RAID1 but in your case thats a
lot of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and made the
important point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <moacirferreira(a)hotmail.com>
wrote:
Hi Colin,
I am in Portugal, so sorry for this late response. It is quite confusing
for me, please consider:
1* - *What if the RAID is done by the server's disk controller, not by
software?
2 - For JBOD I am just using gdeploy to deploy it. However, I am not
using the oVirt node GUI to do this.
3 - As the VM .qcow2 files are quite big, tiering would only help if made
by an intelligent system that uses SSD for chunks of data not for the
entire .qcow2 file. But I guess this is a problem everybody else has. So,
Do you know how tiering works in Gluster?
4 - I am putting the OS on the first disk. However, would you do
differently?
Moacir
------------------------------
*From:* Colin Coe <colin.coe(a)gmail.com>
*Sent:* Monday, August 7, 2017 4:48 AM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org
*Subject:* Re: [ovirt-users] Good practices
1) RAID5 may be a performance hit-
2) I'd be inclined to do this as JBOD by creating a distributed disperse
volume on each server. Something like
echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e
"server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
3) I think the above.
4) Gluster does support tiering, but IIRC you'd need the same number of
SSD as spindle drives. There may be another way to use the SSD as a fast
cache.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <
moacirferreira(a)hotmail.com> wrote:
> I am willing to assemble a oVirt "pod", made of 3 servers, each with 2
> CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
> GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
> a dual 10Gb NIC. So my intention is to create a loop like a server triangle
> using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
> move VMs around the pod (east /west traffic) while using the 10Gb
> interfaces for giving services to the outside world (north/south traffic).
>
>
> This said, my first question is: How should I deploy GlusterFS in such
> oVirt scenario? My questions are:
>
>
> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
> then create a GlusterFS using them?
>
> 2 - Instead, should I create a JBOD array made of all server's disks?
>
> 3 - What is the best Gluster configuration to provide for HA while not
> consuming too much disk space?
>
> 4 - Does a oVirt hypervisor pod like I am planning to build, and the
> virtualization environment, benefits from tiering when using a SSD disk?
> And yes, will Gluster do it by default or I have to configure it to do so?
>
>
> At the bottom line, what is the good practice for using GlusterFS in
> small pods for enterprises?
>
>
> You opinion/feedback will be really appreciated!
>
> Moacir
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
4:03 p.m.
--_000_VI1P190MB02854F23232C41E2AD3C7223C8B50VI1P190MB0285EURP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi Colin,
Take a look on Devin's response. Also, read the doc he shared that gives so=
me hints on how to deploy Gluster.
It is more like that if you want high-performance you should have the brick=
s created as RAID (5 or 6) by the server's disk controller and them assembl=
e a JBOD GlusterFS. The attached document is Gluster specific and not for o=
Virt. But at this point I think that having SSD will not be a plus as using=
the RAID controller Gluster will not be aware of the SSD. Regarding the OS=
, my idea is to have a RAID 1, made of 2 low cost HDDs, to install it.
So far, based on the information received I should create a single RAID 5 o=
r 6 on each server and then use this disk as a brick to create my Gluster c=
luster, made of 2 replicas + 1 arbiter. What is new for me is the detail th=
at the arbiter does not need a lot of space as it only keeps meta data.
Thanks for your response!
Moacir
________________________________
From: Colin Coe <colin.coe(a)gmail.com Sent:
Monday, August 7, 2017 12:41 PM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Hi
I just thought that you'd do hardware RAID if you had the controller or JBO=
D if you didn't. In hindsight, a server with 40Gbps NICs is pretty likely =
to have a hardware RAID controller. I've never done JBOD with hardware RAI=
D. I think having a single gluster brick on hardware JBOD would be riskier=
than multiple bricks, each on a single disk, but thats not based on anythi=
ng other than my prejudices.
I thought gluster tiering was for the most frequently accessed files, in wh=
ich case all the VMs disks would end up in the hot tier. However, I have b=
een wrong before...
I just wanted to know where the OS was going as I didn't see it mentioned i=
n the OP. Normally, I'd have the OS on a RAID1 but in your case thats a lo=
t of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and made the im=
portant point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <moacirferreira(a)hotmail.com=
<mailto:moacirferreira@hotmail.com>> wrote:
Hi Colin,
I am in Portugal, so sorry for this late response. It is quite confusing fo=
r me, please consider:
1 - What if the RAID is done by the server's disk controller, not by softwa=
re?
2 - For JBOD I am just using gdeploy to deploy it. However, I am not using =
the oVirt node GUI to do this.
3 - As the VM .qcow2 files are quite big, tiering would only help if made b=
y an intelligent system that uses SSD for chunks of data not for the entire=
.qcow2 file. But I guess this is a problem everybody else has. So, Do you =
know how tiering works in Gluster?
4 - I am putting the OS on the first disk. However, would you do differentl=
y?
Moacir
________________________________
From: Colin Coe <colin.coe@gmail.com<mailto:colin.coe@gmail.com> Sent: Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: users@ovirt.org<mailto:users@ovirt.org Subject:
Re: [ovirt-users] Good practices
1) RAID5 may be a performance hit-
2) I'd be inclined to do this as JBOD by creating a distributed disperse vo=
lume on each server. Something like
echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERV=
ER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
3) I think the above.
4) Gluster does support tiering, but IIRC you'd need the same number of SSD=
as spindle drives. There may be another way to use the SSD as a fast cach=
e.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_VI1P190MB02854F23232C41E2AD3C7223C8B50VI1P190MB0285EURP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Hi Colin,</p
<p><br </p
<p>Take a look on Devin's response. Also, read the doc he shared that
gives=
some hints on how to deploy Gluster.</p
<p><br </p
<p>It is more like that if you want high-performance you should have the br=
icks created as RAID (5 or 6) by the server's disk controller and them=
assemble a JBOD GlusterFS. The attached document is Gluster specific and n=
ot for oVirt. But at this point I think
that having SSD will not be a plus as using the RAID controller Gluster wi=
ll not be aware of the SSD. Regarding the OS, my idea is to have a RAID 1, =
made of 2 low cost HDDs, to install it.</p
<p><br </p
<p>So far, based on the information received I should create
a si=
ngle RAID 5 or 6 on each server and then use this disk as a brick to create=
my Gluster cluster, made of 2 replicas + 1 arbiter. What is new for me=
is the detail that the arbiter does not need
a lot of space as it only keeps meta data.</p
<p><br </p
<p>Thanks for your response!<br
</p Moacir<br <br <div style=3D"color: rgb(49,
55, 57);" <hr tabindex=3D"-1"
style=3D"display:inline-block; width:98%" <div
id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Colin Coe <colin=
.coe(a)gmail.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 12:41 PM<br <b>To:</b> Moacir
Ferreira<br <b>Cc:</b>
users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
dir=3D"ltr">Hi
<div><br </div <div>I just thought that you'd do hardware RAID if
you had the controller o=
r JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pret=
ty likely to have a hardware RAID controller. I've never done JBOD wi=
th hardware RAID. I think having a single
gluster brick on hardware JBOD would be riskier than multiple bricks, each=
on a single disk, but thats not based on anything other than my prejudices=
.</div <div><br </div
<div>I thought gluster tiering was for the most frequently accessed files, =
in which case all the VMs disks would end up in the hot tier. However=
, I have been wrong before...</div
<div><br </div <div>I just wanted to know where the OS was going as I
didn't see it mentio=
ned in the OP. Normally, I'd have the OS on a RAID1 but in your case =
thats a lot of wasted disk.</div
<div><br </div <div>Honestly, I think Yaniv's answer was far better
than my own and made t=
he important point about having an arbiter. </div
<div><br </div <div>Thanks</div
</div <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira =
<span dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira(a)hotmail.com</a>&gt;</span> wrote:<br <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_4831886883305672718divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p>Hi Colin,</p
<p><br </p
<p><span>I am in Portugal</span>, so sorry for this late response.
It is qu=
ite confusing for me, please consider:</p
<p></p <div><b><br </b>1<b> - </b>What if the RAID is done by
the server's disk controller, no=
t by software?</div <br
<p></p <p>2 -<b> </b>For
JBOD I am just using gdeploy to deploy it. However, I am =
not using the oVirt node GUI to do this.</p
<p><br </p
<p>3 -<b> </b>As the VM .qcow2 files are quite big,
tiering would only=
help if made by an intelligent system that uses SSD for chunks of data not=
for the entire .qcow2 file. But I guess this is a problem everybody else h=
as. So, Do you know how tiering works
in Gluster?<br </p
<p><br </p
<p>4 - I am putting the OS on the first disk. However, would you do
di=
fferently?<br </p
<p><br </p
Moacir<br <br <div
style=3D"color:rgb(49,55,57)" <hr
style=3D"display:inline-block; width:98%" <div
id=3D"m_4831886883305672718divRplyFwdMsg" dir=3D"ltr"><font
style=3D"f=
ont-size:11pt" face=3D"Calibri, sans-serif"
color=3D"#000000"><b>From:</b> =
Colin Coe <<a href=3D"mailto:colin.coe@gmail.com"
target=3D"_blank">coli=
n.coe(a)gmail.com</a>&gt;<br
<b>Sent:</b> Monday, August 7, 2017 4:48 AM<br <b>To:</b> Moacir Ferreira<br <b>Cc:</b> <a
href=3D"mailto:users@ovirt.org" target=3D"_blank">users@ovirt=
.org</a><br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
dir=3D"ltr">1) RAID5 may be a performance hit- <br <div><br
</div <div><span
class=3D"">2) I'd be inclined to do this as JBOD by creating a d=
istributed disperse volume on each server. Something like
<div><br </div <div>echo gluster volume create dispersevol disperse-data
5 redundancy 2 \<=
/div <div>$(for SERVER in a b c; do for BRICK in $(seq 1 5);
do echo -e "se=
rver${SERVER}:/brick/brick-<wbr>${SERVER}${BRICK}/brick \c"; done; don=
e)</div <div><br </div
</span <div>3) I think the above.
<b></b></div <span class=3D"" <div><br
</div <div>4) Gluster does support
tiering, but IIRC you'd need the same number o=
f SSD as spindle drives. There may be another way to use the SSD as a=
fast cache. </div <div><br </div
<div>Where are you putting the OS?</div
<div><br </div <div>Hope I understood the question...</div <div><br
</div <div>Thanks</div </span></div
</div <span class=3D"" <div class=3D"gmail_extra"><br <div class=3D"gmail_quote">On Sun, Aug 6, 2017
at 10:49 PM, Moacir Ferreira=
<span dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira(a)hotmail.com</a>&gt;</span> wrote:<br <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_4831886883305672718m_2460985691746498322divtagdefaultwrapper" =
dir=3D"ltr" style=3D"font-size:12pt; color:#000000;
font-family:Calibri,Hel=
vetica,sans-serif" <p><span>I am willing to
assemble a oVirt "pod", made of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br <span></span></p <p>This said, my first question is: How should I deploy
GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <span
class=3D"m_4831886883305672718HOEnZb"><font
color=3D"#888888" <p>Moacir<br </p
</font></span></div
</div <br
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org"
target=3D"_blank">Users(a)ovirt.org</a><br=
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/...
br <br </blockquote </div
<br </div </span></div
</div </div </div
</blockquote </div <br </div </div
</div </div </body
</html
--_000_VI1P190MB02854F23232C41E2AD3C7223C8B50VI1P190MB0285EURP_--
5:08 a.m.
Moacir, I understand that if you do this type of configuration you will be
severely impacted on storage performance, specially for writes. Even if you
have a Hardware RAID Controller with Writeback cache you will have a
significant performance penalty and may not fully use all the resources you
mentioned you have.
Fernando
2017-08-07 10:03 GMT-03:00 Moacir Ferreira <moacirferreira(a)hotmail.com>:
Hi Colin,
Take a look on Devin's response. Also, read the doc he shared that gives
some hints on how to deploy Gluster.
It is more like that if you want high-performance you should have the
bricks created as RAID (5 or 6) by the server's disk controller and them
assemble a JBOD GlusterFS. The attached document is Gluster specific and
not for oVirt. But at this point I think that having SSD will not be a plus
as using the RAID controller Gluster will not be aware of the SSD.
Regarding the OS, my idea is to have a RAID 1, made of 2 low cost HDDs, to
install it.
So far, based on the information received I should create a single RAID 5
or 6 on each server and then use this disk as a brick to create my Gluster
cluster, made of 2 replicas + 1 arbiter. What is new for me is the detail
that the arbiter does not need a lot of space as it only keeps meta data.
Thanks for your response!
Moacir
------------------------------
*From:* Colin Coe <colin.coe(a)gmail.com>
*Sent:* Monday, August 7, 2017 12:41 PM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org
*Subject:* Re: [ovirt-users] Good practices
Hi
I just thought that you'd do hardware RAID if you had the controller or
JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pretty
likely to have a hardware RAID controller. I've never done JBOD with
hardware RAID. I think having a single gluster brick on hardware JBOD
would be riskier than multiple bricks, each on a single disk, but thats not
based on anything other than my prejudices.
I thought gluster tiering was for the most frequently accessed files, in
which case all the VMs disks would end up in the hot tier. However, I have
been wrong before...
I just wanted to know where the OS was going as I didn't see it mentioned
in the OP. Normally, I'd have the OS on a RAID1 but in your case thats a
lot of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and made the
important point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <
moacirferreira(a)hotmail.com> wrote:
> Hi Colin,
>
>
> I am in Portugal, so sorry for this late response. It is quite confusing
> for me, please consider:
>
>
> 1* - *What if the RAID is done by the server's disk controller, not by
> software?
>
> 2 - For JBOD I am just using gdeploy to deploy it. However, I am not
> using the oVirt node GUI to do this.
>
>
> 3 - As the VM .qcow2 files are quite big, tiering would only help if
> made by an intelligent system that uses SSD for chunks of data not for the
> entire .qcow2 file. But I guess this is a problem everybody else has. So,
> Do you know how tiering works in Gluster?
>
>
> 4 - I am putting the OS on the first disk. However, would you do
> differently?
>
>
> Moacir
>
> ------------------------------
> *From:* Colin Coe <colin.coe(a)gmail.com>
> *Sent:* Monday, August 7, 2017 4:48 AM
> *To:* Moacir Ferreira
> *Cc:* users(a)ovirt.org
> *Subject:* Re: [ovirt-users] Good practices
>
> 1) RAID5 may be a performance hit-
>
> 2) I'd be inclined to do this as JBOD by creating a distributed disperse
> volume on each server. Something like
>
> echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
> $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e
> "server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
>
> 3) I think the above.
>
> 4) Gluster does support tiering, but IIRC you'd need the same number of
> SSD as spindle drives. There may be another way to use the SSD as a fast
> cache.
>
> Where are you putting the OS?
>
> Hope I understood the question...
>
> Thanks
>
> On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <
> moacirferreira(a)hotmail.com> wrote:
>
>> I am willing to assemble a oVirt "pod", made of 3 servers, each with 2
>> CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
>> GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
>> a dual 10Gb NIC. So my intention is to create a loop like a server triangle
>> using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
>> move VMs around the pod (east /west traffic) while using the 10Gb
>> interfaces for giving services to the outside world (north/south traffic).
>>
>>
>> This said, my first question is: How should I deploy GlusterFS in such
>> oVirt scenario? My questions are:
>>
>>
>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
>> then create a GlusterFS using them?
>>
>> 2 - Instead, should I create a JBOD array made of all server's disks?
>>
>> 3 - What is the best Gluster configuration to provide for HA while not
>> consuming too much disk space?
>>
>> 4 - Does a oVirt hypervisor pod like I am planning to build, and the
>> virtualization environment, benefits from tiering when using a SSD disk?
>> And yes, will Gluster do it by default or I have to configure it to do so?
>>
>>
>> At the bottom line, what is the good practice for using GlusterFS in
>> small pods for enterprises?
>>
>>
>> You opinion/feedback will be really appreciated!
>>
>> Moacir
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:16 a.m.
Le 8 août 2017 à 04:08, FERNANDO FREDIANI
<fernando.frediani(a)upx.com> a écrit :
Even if you have a Hardware RAID Controller with Writeback cache you
will have a significant performance penalty and may not fully use all the resources you
mentioned you have.
Nope again,from my experience with HP Smart Array and write back cache, write, that goes
in the cache, are even faster that read that must goes to the disks. of course if the
write are too fast and to big, they will over overflow the cache. But on todays controller
they are multi-gigabyte cache, you must write a lot to fill them. And if you can afford
40Gb card, you can afford decent controller.
9:50 a.m.
On Tue, Aug 8, 2017 at 9:16 AM, Fabrice Bacchella <
fabrice.bacchella(a)orange.fr> wrote:
> Le 8 août 2017 à 04:08, FERNANDO FREDIANI <fernando.frediani(a)upx.com> a
écrit :
> Even if you have a Hardware RAID Controller with Writeback cache you
will have a significant performance penalty and may not fully use all the
resources you mentioned you have.
>
Nope again,from my experience with HP Smart Array and write back cache,
write, that goes in the cache, are even faster that read that must goes to
the disks. of course if the write are too fast and to big, they will over
overflow the cache. But on todays controller they are multi-gigabyte cache,
you must write a lot to fill them. And if you can afford 40Gb card, you can
afford decent controller.
The last sentence raises an excellent point: balance your resources. Don't
spend a fortune on one component while another will end up being your
bottleneck.
Storage is usually the slowest link in the chain. I personally believe that
spending the money on NVMe drives makes more sense than 40Gb (except [1],
which is suspiciously cheap!)
Y.
[1] http://a.co/4hsCTqG
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:34 a.m.
--Apple-Mail=_88F0455F-3BE8-4421-89AF-B80F9A90AAF2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Le 8 ao=C3=BBt 2017 =C3=A0 08:50, Yaniv Kaul <ykaul(a)redhat.com>
a =
=C3=A9crit :
=20
Storage is usually the slowest link in the chain. I personally
believe =
that spending the money on NVMe drives makes more sense than 40Gb =
(except [1], which is suspiciously cheap!)
=20
> Y.
> [1] http://a.co/4hsCTqG <http://a.co/4hsCTqG>
=
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=3Demr_na-c0437407=
8
It's supported on old Gen8 servers (G10 is comming). It must be coming =
from an attic.
--Apple-Mail=_88F0455F-3BE8-4421-89AF-B80F9A90AAF2
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">Le 8 ao=C3=BBt 2017 =C3=A0 08:50, Yaniv Kaul <<a =
href=3D"mailto:ykaul@redhat.com"
class=3D"">ykaul(a)redhat.com</a>&gt; a =
=C3=A9crit :</div><br class=3D"Apple-interchange-newline"><div =
class=3D""><div dir=3D"ltr" class=3D""><div
class=3D"gmail_extra"><div =
class=3D"gmail_quote"></div></div></div></div></blockquote><br
=
class=3D""><blockquote type=3D"cite"
class=3D""><div class=3D""><div =
dir=3D"ltr" class=3D""><div
class=3D"gmail_extra"><div =
class=3D"gmail_quote"><div class=3D"">Storage is usually the
slowest =
link in the chain. I personally believe that spending the money on NVMe =
drives makes more sense than 40Gb (except [1], which is suspiciously =
cheap!)</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Y.</div><div
class=3D"">[1] <span =
style=3D"color:rgb(17,17,17);font-family:"Amazon =
Ember",Arial,sans-serif;font-size:13px" class=3D""><a =
href=3D"http://a.co/4hsCTqG" =
class=3D"">http://a.co/4hsCTqG</a></span></div></div></div></div></div></b=
lockquote><br class=3D""></div><div><br
class=3D""></div><div><a =
href=3D"http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=3De...
c04374078" =
class=3D"">http://h20564.www2.hpe.com/hpsc/doc/public/displa...
na-c04374078</a></div><div><br
class=3D""></div><div>It's supported on =
old Gen8 servers (G10 is comming). It must be coming from an =
attic.</div><div><br class=3D""></div><br
class=3D""></body></html>=
--Apple-Mail=_88F0455F-3BE8-4421-89AF-B80F9A90AAF2--
4:24 p.m.
That's something on the way RAID works, regardless what most
'super-ultra' powerfull hardware controller you may have. RAID 5 or 6
will never have the same write performance as a RAID 10 o 0 for example.
Writeback caches can deal with bursts well but they have a limit
therefore there will always be a penalty compared to what else you could
have.
If you have a continuous stream of data (a big VM deployment or a large
data copy) there will be a continuous write and that will likely fill up
the cache making the disks underneath the bottleneck.
That's why on some other scenarios, like ZFS people have multiple groups
of RAID 6 (called RAIDZ2) so it improves the write speeds for these type
of scenarios.
In the scenario given in this thread with just 3 servers, each with a
RAID 6 there will be a bare limit on the write performance specially for
streammed data for most powerfull your hardware controller can do
write-back.
Also I agree the 40Gb NICs may not be used fully and 10Gb can do the job
well, but if they were available at the begining, why not use them.
Fernando
On 08/08/2017 03:16, Fabrice Bacchella wrote:
> Le 8 août 2017 à 04:08, FERNANDO FREDIANI
<fernando.frediani(a)upx.com> a écrit :
> Even if you have a Hardware RAID Controller with Writeback cache you will have a
significant performance penalty and may not fully use all the resources you mentioned you
have.
>
Nope again,from my experience with HP Smart Array and write back cache, write, that goes
in the cache, are even faster that read that must goes to the disks. of course if the
write are too fast and to big, they will over overflow the cache. But on todays controller
they are multi-gigabyte cache, you must write a lot to fill them. And if you can afford
40Gb card, you can afford decent controller.
4:39 p.m.
On tis, 2017-08-08 at 10:24 -0300, FERNANDO FREDIANI wrote:
That's something on the way RAID works, regardless what most
'super-ultra' powerfull hardware controller you may have. RAID 5 or
6
will never have the same write performance as a RAID 10 o 0 for
example.
Writeback caches can deal with bursts well but they have a limit
therefore there will always be a penalty compared to what else you
could
have.
If you have a continuous stream of data (a big VM deployment or a
large
data copy) there will be a continuous write and that will likely fill
up
the cache making the disks underneath the bottleneck.
That's why on some other scenarios, like ZFS people have multiple
groups
of RAID 6 (called RAIDZ2) so it improves the write speeds for these
type
of scenarios.
Just pointing out that it is commonly known as RAID 60, outside of the
ZFS lingo:
https://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_60
/K
In the scenario given in this thread with just 3 servers, each with
a
RAID 6 there will be a bare limit on the write performance specially
for
streammed data for most powerfull your hardware controller can do
write-back.
Also I agree the 40Gb NICs may not be used fully and 10Gb can do the
job
well, but if they were available at the begining, why not use them.
Fernando
On 08/08/2017 03:16, Fabrice Bacchella wrote:
>
> >
> > Le 8 août 2017 à 04:08, FERNANDO FREDIANI <fernando.frediani@upx.
> > com> a écrit :
> > Even if you have a Hardware RAID Controller with Writeback cache
> > you will have a significant performance penalty and may not fully
> > use all the resources you mentioned you have.
> >
> Nope again,from my experience with HP Smart Array and write back
> cache, write, that goes in the cache, are even faster that read
> that must goes to the disks. of course if the write are too fast
> and to big, they will over overflow the cache. But on todays
> controller they are multi-gigabyte cache, you must write a lot to
> fill them. And if you can afford 40Gb card, you can afford decent
> controller.
>
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:48 p.m.
Le 8 août 2017 à 15:24, FERNANDO FREDIANI
<fernando.frediani(a)upx.com> a écrit :
That's something on the way RAID works, regardless what most 'super-ultra'
powerfull hardware controller you may have. RAID 5 or 6 will never have the same write
performance as a RAID 10 o 0 for example. Writeback caches can deal with bursts well but
they have a limit therefore there will always be a penalty compared to what else you could
have.
Hardware RAID5/6 can have better performance with quite common hardware that software
RAID0. I have seen many time on on even old servers that write latency (hitting the cache)
was smaller that read latency that was going directly to the disk. I'm not talking
about 'super-ultra' powerfull hardware. An HP Smart Array P440ar with 2 GB flash
is sell at 560€, public price. Not cheap, but not ultra powerfull.
It's now a matter of identifying the bootle neck, and how much money you can throw at
it.
5:44 p.m.
Fernando,
I agree that RAID is not required here by common sense. The only point to setup RAID is a
lack of manageability of GlusterFS. So you just buy manageability for extra hardware cost
and write performance in some scenarios. That is it.
On 08/08/2017, 16:24, "users-bounces(a)ovirt.org on behalf of FERNANDO FREDIANI"
<users-bounces(a)ovirt.org on behalf of fernando.frediani(a)upx.com> wrote:
That's something on the way RAID works, regardless what most
'super-ultra' powerfull hardware controller you may have. RAID 5 or 6
will never have the same write performance as a RAID 10 o 0 for example.
Writeback caches can deal with bursts well but they have a limit
therefore there will always be a penalty compared to what else you could
have.
If you have a continuous stream of data (a big VM deployment or a large
data copy) there will be a continuous write and that will likely fill up
the cache making the disks underneath the bottleneck.
That's why on some other scenarios, like ZFS people have multiple groups
of RAID 6 (called RAIDZ2) so it improves the write speeds for these type
of scenarios.
In the scenario given in this thread with just 3 servers, each with a
RAID 6 there will be a bare limit on the write performance specially for
streammed data for most powerfull your hardware controller can do
write-back.
Also I agree the 40Gb NICs may not be used fully and 10Gb can do the job
well, but if they were available at the begining, why not use them.
Fernando
On 08/08/2017 03:16, Fabrice Bacchella wrote:
> Le 8 août 2017 à 04:08, FERNANDO FREDIANI
<fernando.frediani(a)upx.com> a écrit :
> Even if you have a Hardware RAID Controller with Writeback cache you will have a
significant performance penalty and may not fully use all the resources you mentioned you
have.
>
Nope again,from my experience with HP Smart Array and write back cache, write, that goes
in the cache, are even faster that read that must goes to the disks. of course if the
write are too fast and to big, they will over overflow the cache. But on todays controller
they are multi-gigabyte cache, you must write a lot to fill them. And if you can afford
40Gb card, you can afford decent controller.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
12:09 p.m.
--_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Fernando,
Let's see what people say... But this is what I understood Red Hat says is =
the best performance model. This is the main reason to open this discussion=
because as long as I can see, some of you in the community, do not agree.
But when I think about a "distributed file system", that can make any numbe=
r of copies you want, it does not make sense using a RAIDed brick, what it =
makes sense is to use JBOD.
Moacir
________________________________
From: fernando.frediani(a)upx.com.br <fernando.frediani(a)upx.com.br> on behalf=
of FERNANDO FREDIANI <fernando.frediani(a)upx.com Sent:
Tuesday, August 8, 2017 3:08 AM
To: Moacir Ferreira
Cc: Colin Coe; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir, I understand that if you do this type of configuration you will be =
severely impacted on storage performance, specially for writes. Even if you=
have a Hardware RAID Controller with Writeback cache you will have a signi=
ficant performance penalty and may not fully use all the resources you ment=
ioned you have.
Fernando
2017-08-07 10:03 GMT-03:00 Moacir Ferreira <moacirferreira(a)hotmail.com<mail=
to:moacirferreira@hotmail.com>>:
Hi Colin,
Take a look on Devin's response. Also, read the doc he shared that gives so=
me hints on how to deploy Gluster.
It is more like that if you want high-performance you should have the brick=
s created as RAID (5 or 6) by the server's disk controller and them assembl=
e a JBOD GlusterFS. The attached document is Gluster specific and not for o=
Virt. But at this point I think that having SSD will not be a plus as using=
the RAID controller Gluster will not be aware of the SSD. Regarding the OS=
, my idea is to have a RAID 1, made of 2 low cost HDDs, to install it.
So far, based on the information received I should create a single RAID 5 o=
r 6 on each server and then use this disk as a brick to create my Gluster c=
luster, made of 2 replicas + 1 arbiter. What is new for me is the detail th=
at the arbiter does not need a lot of space as it only keeps meta data.
Thanks for your response!
Moacir
________________________________
From: Colin Coe <colin.coe@gmail.com<mailto:colin.coe@gmail.com> Sent: Monday, August 7, 2017 12:41 PM
To: Moacir Ferreira
Cc: users@ovirt.org<mailto:users@ovirt.org Subject:
Re: [ovirt-users] Good practices
Hi
I just thought that you'd do hardware RAID if you had the controller or JBO=
D if you didn't. In hindsight, a server with 40Gbps NICs is pretty likely =
to have a hardware RAID controller. I've never done JBOD with hardware RAI=
D. I think having a single gluster brick on hardware JBOD would be riskier=
than multiple bricks, each on a single disk, but thats not based on anythi=
ng other than my prejudices.
I thought gluster tiering was for the most frequently accessed files, in wh=
ich case all the VMs disks would end up in the hot tier. However, I have b=
een wrong before...
I just wanted to know where the OS was going as I didn't see it mentioned i=
n the OP. Normally, I'd have the OS on a RAID1 but in your case thats a lo=
t of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and made the im=
portant point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <moacirferreira(a)hotmail.com=
<mailto:moacirferreira@hotmail.com>> wrote:
Hi Colin,
I am in Portugal, so sorry for this late response. It is quite confusing fo=
r me, please consider:
1 - What if the RAID is done by the server's disk controller, not by softwa=
re?
2 - For JBOD I am just using gdeploy to deploy it. However, I am not using =
the oVirt node GUI to do this.
3 - As the VM .qcow2 files are quite big, tiering would only help if made b=
y an intelligent system that uses SSD for chunks of data not for the entire=
.qcow2 file. But I guess this is a problem everybody else has. So, Do you =
know how tiering works in Gluster?
4 - I am putting the OS on the first disk. However, would you do differentl=
y?
Moacir
________________________________
From: Colin Coe <colin.coe@gmail.com<mailto:colin.coe@gmail.com> Sent: Monday, August 7, 2017 4:48 AM
To: Moacir Ferreira
Cc: users@ovirt.org<mailto:users@ovirt.org Subject:
Re: [ovirt-users] Good practices
1) RAID5 may be a performance hit-
2) I'd be inclined to do this as JBOD by creating a distributed disperse vo=
lume on each server. Something like
echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERV=
ER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
3) I think the above.
4) Gluster does support tiering, but IIRC you'd need the same number of SSD=
as spindle drives. There may be another way to use the SSD as a fast cach=
e.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Fernando,</p
<p><br </p
<p>Let's see what people say... But this is what I understood Red Hat says
=
is the best performance model. This is the main reason to open this discuss=
ion because as long as I can see, some of you in the community, do not agre=
e.<br </p
<br <p>But when I think about a
"distributed file system", that can m=
ake any number of copies you want, it does not make sense using a RAIDed br=
ick, what it makes sense is to use JBOD.</p
<p><br </p
<p>Moacir<br </p
<br <div style=3D"color: rgb(49,
55, 57);" <hr tabindex=3D"-1"
style=3D"display:inline-block; width:98%" <div
id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
fernando.frediani@u=
px.com.br &lt;fernando.frediani(a)upx.com.br&gt; on behalf of FERNANDO FREDIA=
NI &lt;fernando.frediani(a)upx.com&gt;<br
<b>Sent:</b> Tuesday, August 8, 2017 3:08 AM<br <b>To:</b> Moacir Ferreira<br <b>Cc:</b> Colin Coe; users(a)ovirt.org<br <b>Subject:</b> Re: [ovirt-users] Good
practices</font <div> </div </div
<div <div dir=3D"ltr" <div>Moacir, I understand that if you do this type of
configuration you wil=
l be severely impacted on storage performance, specially for writes. Even i=
f you have a Hardware RAID Controller with Writeback cache you will have a =
significant performance penalty
and may not fully use all the resources you mentioned you have.<br <br </div Fernando<br
</div <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">2017-08-07 10:03 GMT-03:00 Moacir Ferreira <span=
dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira@hotmail.com</a>></span>:<br
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;
border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_1874859859842763104divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p>Hi Colin,</p
<p><br </p
<p>Take a look on Devin's response. Also, read the doc he shared that
gives=
some hints on how to deploy Gluster.</p
<p><br </p
<p>It is more like that if you want high-performance you should have the br=
icks created as RAID (5 or 6) by the server's disk controller and them=
assemble a JBOD GlusterFS. The attached document is Gluster specific and n=
ot for oVirt. But at this point I think
that having SSD will not be a plus as using the RAID controller Gluster wi=
ll not be aware of the SSD. Regarding the OS, my idea is to have a RAID 1, =
made of 2 low cost HDDs, to install it.</p
<p><br </p
<p>So far, based on the information received I should create
a si=
ngle RAID 5 or 6 on each server and then use this disk as a brick to create=
my Gluster cluster, made of 2 replicas + 1 arbiter. What is new for me=
is the detail that the arbiter does not need
a lot of space as it only keeps meta data.</p
<p><br </p
<p>Thanks for your response!<br
</p Moacir<br <br <div
style=3D"color:rgb(49,55,57)" <hr
style=3D"display:inline-block; width:98%" <div
id=3D"m_1874859859842763104divRplyFwdMsg" dir=3D"ltr"><font
style=3D"f=
ont-size:11pt" face=3D"Calibri, sans-serif"
color=3D"#000000"><b>From:</b> =
Colin Coe <<a href=3D"mailto:colin.coe@gmail.com"
target=3D"_blank">coli=
n.coe(a)gmail.com</a>&gt;<br
<b>Sent:</b> Monday, August 7, 2017 12:41 PM
<div <div class=3D"h5"><br <b>To:</b> Moacir Ferreira<br <b>Cc:</b> <a
href=3D"mailto:users@ovirt.org" target=3D"_blank">users@ovirt=
.org</a><br <b>Subject:</b> Re:
[ovirt-users] Good practices</div
</div </font <div> </div
</div <div <div
class=3D"h5" <div <div
dir=3D"ltr">Hi
<div><br </div <div>I just thought that you'd do hardware RAID if
you had the controller o=
r JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pret=
ty likely to have a hardware RAID controller. I've never done JBOD wi=
th hardware RAID. I think having a single
gluster brick on hardware JBOD would be riskier than multiple bricks, each=
on a single disk, but thats not based on anything other than my prejudices=
.</div <div><br </div
<div>I thought gluster tiering was for the most frequently accessed files, =
in which case all the VMs disks would end up in the hot tier. However=
, I have been wrong before...</div
<div><br </div <div>I just wanted to know where the OS was going as I
didn't see it mentio=
ned in the OP. Normally, I'd have the OS on a RAID1 but in your case =
thats a lot of wasted disk.</div
<div><br </div <div>Honestly, I think Yaniv's answer was far better
than my own and made t=
he important point about having an arbiter. </div
<div><br </div <div>Thanks</div
</div <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira =
<span dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira(a)hotmail.com</a>&gt;</span> wrote:<br <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_1874859859842763104m_4831886883305672718divtagdefaultwrapper" =
dir=3D"ltr" style=3D"font-size:12pt; color:#000000;
font-family:Calibri,Hel=
vetica,sans-serif" <p>Hi Colin,</p <p><br
</p <p><span>I am in
Portugal</span>, so sorry for this late response. It is qu=
ite confusing for me, please consider:</p
<p></p <div><b><br </b>1<b> - </b>What if the RAID is done by
the server's disk controller, no=
t by software?</div <br
<p></p <p>2 -<b> </b>For
JBOD I am just using gdeploy to deploy it. However, I am =
not using the oVirt node GUI to do this.</p
<p><br </p
<p>3 -<b> </b>As the VM .qcow2 files are quite big,
tiering would only=
help if made by an intelligent system that uses SSD for chunks of data not=
for the entire .qcow2 file. But I guess this is a problem everybody else h=
as. So, Do you know how tiering works
in Gluster?<br </p
<p><br </p
<p>4 - I am putting the OS on the first disk. However, would you do
di=
fferently?<br </p
<p><br </p
Moacir<br <br <div
style=3D"color:rgb(49,55,57)" <hr
style=3D"display:inline-block; width:98%" <div
id=3D"m_1874859859842763104m_4831886883305672718divRplyFwdMsg" dir=3D"=
ltr"><font style=3D"font-size:11pt" face=3D"Calibri,
sans-serif" color=3D"#=
000000"><b>From:</b> Colin Coe <<a
href=3D"mailto:colin.coe@gmail.com" t=
arget=3D"_blank">colin.coe(a)gmail.com</a>&gt;<br <b>Sent:</b> Monday, August 7, 2017 4:48
AM<br <b>To:</b> Moacir
Ferreira<br <b>Cc:</b> <a
href=3D"mailto:users@ovirt.org" target=3D"_blank">users@ovirt=
.org</a><br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
dir=3D"ltr">1) RAID5 may be a performance hit- <br <div><br
</div <div><span>2) I'd be
inclined to do this as JBOD by creating a distributed =
disperse volume on each server. Something like
<div><br </div <div>echo gluster volume create dispersevol disperse-data
5 redundancy 2 \<=
/div <div>$(for SERVER in a b c; do for BRICK in $(seq 1 5);
do echo -e "se=
rver${SERVER}:/brick/brick-<wbr>${SERVER}${BRICK}/brick \c"; done; don=
e)</div <div><br </div
</span <div>3) I think the above.
<b></b></div <span <div><br
</div <div>4) Gluster does support
tiering, but IIRC you'd need the same number o=
f SSD as spindle drives. There may be another way to use the SSD as a=
fast cache. </div <div><br </div
<div>Where are you putting the OS?</div
<div><br </div <div>Hope I understood the question...</div <div><br
</div <div>Thanks</div </span></div
</div <span <div class=3D"gmail_extra"><br <div class=3D"gmail_quote">On Sun, Aug 6, 2017
at 10:49 PM, Moacir Ferreira=
<span dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira(a)hotmail.com</a>&gt;</span> wrote:<br <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_1874859859842763104m_4831886883305672718m_2460985691746498322d=
ivtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color:#000000; fo=
nt-family:Calibri,Helvetica,sans-serif"
<p><span>I am willing to assemble a oVirt "pod", made
of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br <span></span></p <p>This said, my first question is: How should I deploy
GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <span
class=3D"m_1874859859842763104m_4831886883305672718HOEnZb"><font colo=
r=3D"#888888" <p>Moacir<br </p
</font></span></div
</div <br
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org"
target=3D"_blank">Users(a)ovirt.org</a><br=
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/...
br <br </blockquote </div
<br </div </span></div
</div </div </div
</blockquote </div <br </div </div
</div </div </div
</div </div <br
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/<wbr>mailman/...
br <br </blockquote </div
<br </div </div
</div </div </body
</html
--_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_--
4:31 p.m.
This is a multi-part message in MIME format.
--------------A95CF1C0E4312049A540BB6A
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Exactly Moacir, that is my point.
A proper Distributed FIlesystem should not rely on any type of RAID as
it can make its own redundancy without having to rely on any underneath
layer (look at CEPH). Using RAID may help with management and in certain
scenarios to replace a faulty disk, but at a cost, not cheap by the way.
That's why in terms of resourcing saving, if a replica 3 brings those
issues mentioned it is much worth to have a small arbiter somewhere
instead of wasting a significant amount of disk space.
Fernando
On 08/08/2017 06:09, Moacir Ferreira wrote:
Fernando,
Let's see what people say... But this is what I understood Red Hat
says is the best performance model. This is the main reason to open
this discussion because as long as I can see, some of you in the
community, do not agree.
But when I think about a "distributed file system", that can make any
number of copies you want, it does not make sense using a RAIDed
brick, what it makes sense is to use JBOD.
Moacir
------------------------------------------------------------------------
*From:* fernando.frediani(a)upx.com.br <fernando.frediani(a)upx.com.br> on
behalf of FERNANDO FREDIANI <fernando.frediani(a)upx.com>
*Sent:* Tuesday, August 8, 2017 3:08 AM
*To:* Moacir Ferreira
*Cc:* Colin Coe; users(a)ovirt.org
*Subject:* Re: [ovirt-users] Good practices
Moacir, I understand that if you do this type of configuration you
will be severely impacted on storage performance, specially for
writes. Even if you have a Hardware RAID Controller with Writeback
cache you will have a significant performance penalty and may not
fully use all the resources you mentioned you have.
Fernando
2017-08-07 10:03 GMT-03:00 Moacir Ferreira <moacirferreira(a)hotmail.com
<mailto:moacirferreira@hotmail.com>>:
Hi Colin,
Take a look on Devin's response. Also, read the doc he shared that
gives some hints on how to deploy Gluster.
It is more like that if you want high-performance you should have
the bricks created as RAID (5 or 6) by the server's disk
controller and them assemble a JBOD GlusterFS. The attached
document is Gluster specific and not for oVirt. But at this point
I think that having SSD will not be a plus as using the RAID
controller Gluster will not be aware of the SSD. Regarding the OS,
my idea is to have a RAID 1, made of 2 low cost HDDs, to install it.
So far, based on the information received I should create a single
RAID 5 or 6 on each server and then use this disk as a brick to
create my Gluster cluster, made of 2 replicas + 1 arbiter. What is
new for me is the detail that the arbiter does not need a lot of
space as it only keeps meta data.
Thanks for your response!
Moacir
------------------------------------------------------------------------
*From:* Colin Coe <colin.coe(a)gmail.com <mailto:colin.coe@gmail.com>>
*Sent:* Monday, August 7, 2017 12:41 PM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org <mailto:users@ovirt.org>
*Subject:* Re: [ovirt-users] Good practices
Hi
I just thought that you'd do hardware RAID if you had the
controller or JBOD if you didn't. In hindsight, a server with
40Gbps NICs is pretty likely to have a hardware RAID controller.
I've never done JBOD with hardware RAID. I think having a single
gluster brick on hardware JBOD would be riskier than multiple
bricks, each on a single disk, but thats not based on anything
other than my prejudices.
I thought gluster tiering was for the most frequently accessed
files, in which case all the VMs disks would end up in the hot
tier. However, I have been wrong before...
I just wanted to know where the OS was going as I didn't see it
mentioned in the OP. Normally, I'd have the OS on a RAID1 but in
your case thats a lot of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and
made the important point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira
<moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>>
wrote:
Hi Colin,
I am in Portugal, so sorry for this late response. It is quite
confusing for me, please consider:
*
*1*- *What if the RAID is done by the server's disk
controller, not by software?
2 -**For JBOD I am just using gdeploy to deploy it. However, I
am not using the oVirt node GUI to do this.
3 -**As the VM .qcow2 files are quite big, tiering would only
help if made by an intelligent system that uses SSD for chunks
of data not for the entire .qcow2 file. But I guess this is a
problem everybody else has. So, Do you know how tiering works
in Gluster?
4 - I am putting the OS on the first disk. However, would you
do differently?
Moacir
------------------------------------------------------------------------
*From:* Colin Coe <colin.coe(a)gmail.com
<mailto:colin.coe@gmail.com>>
*Sent:* Monday, August 7, 2017 4:48 AM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org <mailto:users@ovirt.org>
*Subject:* Re: [ovirt-users] Good practices
1) RAID5 may be a performance hit-
2) I'd be inclined to do this as JBOD by creating a
distributed disperse volume on each server. Something like
echo gluster volume create dispersevol disperse-data 5
redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e
"server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c";
done; done)
3) I think the above.
4) Gluster does support tiering, but IIRC you'd need the same
number of SSD as spindle drives. There may be another way to
use the SSD as a fast cache.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira
<moacirferreira(a)hotmail.com
<mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers,
each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K,
1 SSD. The idea is to use GlusterFS to provide HA for the
VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb
NIC. So my intention is to create a loop like a server
triangle using the 40Gb NICs for virtualization files (VMs
.qcow2) access and to move VMs around the pod (east /west
traffic) while using the 10Gb interfaces for giving
services to the outside world (north/south traffic).
This said, my first question is: How should I deploy
GlusterFS in such oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each
oVirt node, and then create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all
server's disks?
3 - What is the best Gluster configuration to provide for
HA while not consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to
build, and the virtualization environment, benefits from
tiering when using a SSD disk? And yes, will Gluster do it
by default or I have to configure it to do so?
At the bottom line, what is the good practice for using
GlusterFS in small pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
--------------A95CF1C0E4312049A540BB6A
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Exactly Moacir, that is my point.</p>
<p><br>
</p>
<p>A proper Distributed FIlesystem should not rely on any type of
RAID as it can make its own redundancy without having to rely on
any underneath layer (look at CEPH). Using RAID may help with
management and in certain scenarios to replace a faulty disk, but
at a cost, not cheap by the way.<br>
That's why in terms of resourcing saving, if a replica 3 brings
those issues mentioned it is much worth to have a small arbiter
somewhere instead of wasting a significant amount of disk space.</p>
<p><br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/08/2017 06:09, Moacir Ferreira
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DB6P190MB02801EA0892B38F503220896C88A0@DB6P190MB0280.EURP190.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
dir="ltr">
<p>Fernando,</p>
<p><br>
</p>
<p>Let's see what people say... But this is what I understood
Red Hat says is the best performance model. This is the main
reason to open this discussion because as long as I can see,
some of you in the community, do not agree.<br>
</p>
<br>
<p>But when I think about a "distributed file system", that can
make any number of copies you want, it does not make sense
using a RAIDed brick, what it makes sense is to use JBOD.</p>
<p><br>
</p>
<p>Moacir<br>
</p>
<br>
<div style="color: rgb(49, 55, 57);">
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
<a class="moz-txt-link-abbreviated"
href="mailto:fernando.frediani@upx.com.br">fernando.frediani@upx.com.br</a>
<a class="moz-txt-link-rfc2396E"
href="mailto:fernando.frediani@upx.com.br"><fernando.frediani@upx.com.br></a>
on behalf of FERNANDO
FREDIANI <a class="moz-txt-link-rfc2396E"
href="mailto:fernando.frediani@upx.com"><fernando.frediani@upx.com></a><br>
<b>Sent:</b> Tuesday, August 8, 2017 3:08 AM<br>
<b>To:</b> Moacir Ferreira<br>
<b>Cc:</b> Colin Coe; <a
class="moz-txt-link-abbreviated"
href="mailto:users@ovirt.org">users@ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] Good practices</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Moacir, I understand that if you do this type of
configuration you will be severely impacted on storage
performance, specially for writes. Even if you have a
Hardware RAID Controller with Writeback cache you will
have a significant performance penalty and may not fully
use all the resources you mentioned you have.<br>
<br>
</div>
Fernando<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2017-08-07 10:03 GMT-03:00 Moacir
Ferreira <span dir="ltr">
<<a href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira@hotmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_1874859859842763104divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>Hi Colin,</p>
<p><br>
</p>
<p>Take a look on Devin's response. Also, read the
doc he shared that gives some hints on how to
deploy Gluster.</p>
<p><br>
</p>
<p>It is more like that if you want
high-performance you should have the bricks
created as RAID (5 or 6) by the server's disk
controller and them assemble a JBOD GlusterFS.
The attached document is Gluster specific and
not for oVirt. But at this point I think that
having SSD will not be a plus as using the RAID
controller Gluster will not be aware of the SSD.
Regarding the OS, my idea is to have a RAID 1,
made of 2 low cost HDDs, to install it.</p>
<p><br>
</p>
<p>So far, based on the information received I
should create a single RAID 5 or 6 on each
server and then use this disk as a brick to
create my Gluster cluster, made of 2 replicas +
1 arbiter. What is new for me is the detail that
the arbiter does not need a lot of space as it
only keeps meta data.</p>
<p><br>
</p>
<p>Thanks for your response!<br>
</p>
Moacir<br>
<br>
<div style="color:rgb(49,55,57)">
<hr style="display:inline-block; width:98%">
<div id="m_1874859859842763104divRplyFwdMsg"
dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
Colin Coe <<a
href="mailto:colin.coe@gmail.com"
target="_blank"
moz-do-not-send="true">colin.coe(a)gmail.com</a>&gt;<br>
<b>Sent:</b> Monday, August 7, 2017 12:41 PM
<div>
<div class="h5"><br>
<b>To:</b> Moacir Ferreira<br>
<b>Cc:</b> <a
href="mailto:users@ovirt.org"
target="_blank"
moz-do-not-send="true">users(a)ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] Good
practices</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="h5">
<div>
<div dir="ltr">Hi
<div><br>
</div>
<div>I just thought that you'd do
hardware RAID if you had the
controller or JBOD if you didn't. In
hindsight, a server with 40Gbps NICs
is pretty likely to have a hardware
RAID controller. I've never done JBOD
with hardware RAID. I think having a
single gluster brick on hardware JBOD
would be riskier than multiple bricks,
each on a single disk, but thats not
based on anything other than my
prejudices.</div>
<div><br>
</div>
<div>I thought gluster tiering was for
the most frequently accessed files, in
which case all the VMs disks would end
up in the hot tier. However, I have
been wrong before...</div>
<div><br>
</div>
<div>I just wanted to know where the OS
was going as I didn't see it mentioned
in the OP. Normally, I'd have the OS
on a RAID1 but in your case thats a
lot of wasted disk.</div>
<div><br>
</div>
<div>Honestly, I think Yaniv's answer
was far better than my own and made
the important point about having an
arbiter. </div>
<div><br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 7,
2017 at 5:56 PM, Moacir Ferreira <span
dir="ltr">
<<a
href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;
border-left:1px #ccc solid;
padding-left:1ex">
<div dir="ltr">
<div
id="m_1874859859842763104m_4831886883305672718divtagdefaultwrapper"
dir="ltr" style="font-size:12pt;
color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>Hi Colin,</p>
<p><br>
</p>
<p><span>I am in
Portugal</span>,
so sorry for this late
response. It is quite
confusing for me, please
consider:</p>
<div><b><br>
</b>1<b> - </b>What if the
RAID is done by the server's
disk controller, not by
software?</div>
<br>
<p>2 -<b> </b>For JBOD I am
just using gdeploy to deploy
it. However, I am not using
the oVirt node GUI to do this.</p>
<p><br>
</p>
<p>3 -<b> </b>As the VM
.qcow2 files are quite big,
tiering would only help if
made by an intelligent system
that uses SSD for chunks of
data not for the entire .qcow2
file. But I guess this is a
problem everybody else has.
So, Do you know how tiering
works in Gluster?<br>
</p>
<p><br>
</p>
<p>4 - I am putting the OS on
the first disk. However, would
you do differently?<br>
</p>
<p><br>
</p>
Moacir<br>
<br>
<div style="color:rgb(49,55,57)">
<hr
style="display:inline-block;
width:98%">
<div
id="m_1874859859842763104m_4831886883305672718divRplyFwdMsg"
dir="ltr"><font
style="font-size:11pt"
color="#000000"
face="Calibri,
sans-serif"><b>From:</b>
Colin Coe <<a
href="mailto:colin.coe@gmail.com"
target="_blank"
moz-do-not-send="true">colin.coe(a)gmail.com</a>&gt;<br>
<b>Sent:</b> Monday,
August 7, 2017 4:48 AM<br>
<b>To:</b> Moacir
Ferreira<br>
<b>Cc:</b> <a
href="mailto:users@ovirt.org"
target="_blank"
moz-do-not-send="true">users(a)ovirt.org</a><br>
<b>Subject:</b> Re:
[ovirt-users] Good
practices</font>
<div> </div>
</div>
<div>
<div dir="ltr">1) RAID5 may
be a performance hit- <br>
<div><br>
</div>
<div><span>2) I'd be
inclined to do this as
JBOD by creating a
distributed disperse
volume on each
server. Something
like
<div><br>
</div>
<div>echo gluster
volume create
dispersevol
disperse-data 5
redundancy 2 \</div>
<div>$(for SERVER in a
b c; do for BRICK in
$(seq 1 5); do echo
-e
"server${SERVER}:/brick/brick-<wbr>${SERVER}${BRICK}/brick
\c"; done; done)</div>
<div><br>
</div>
</span>
<div>3) I think the
above. </div>
<span>
<div><br>
</div>
<div>4) Gluster does
support tiering, but
IIRC you'd need the
same number of SSD
as spindle drives.
There may be another
way to use the SSD
as a fast cache. </div>
<div><br>
</div>
<div>Where are you
putting the OS?</div>
<div><br>
</div>
<div>Hope I understood
the question...</div>
<div><br>
</div>
<div>Thanks</div>
</span></div>
</div>
<span>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Sun, Aug 6, 2017 at
10:49 PM, Moacir
Ferreira <span
dir="ltr">
<<a
href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>&gt;</span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;
border-left:1px #ccc
solid;
padding-left:1ex">
<div dir="ltr">
<div
id="m_1874859859842763104m_4831886883305672718m_2460985691746498322divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;
color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p><span>I am
willing to
assemble a
oVirt "pod",
made of 3
servers, each
with 2 CPU
sockets of 12
cores, 256GB
RAM, 7 HDD
10K, 1 SSD.
The idea is to
use GlusterFS
to provide HA
for the VMs.
The 3 servers
have a dual
40Gb NIC and a
dual 10Gb NIC.
So my
intention is
to create a
loop like a
server
triangle using
the 40Gb NICs
for
virtualization
files (VMs
.qcow2) access
and to move
VMs around the
pod (east
/west traffic)
while using
the 10Gb
interfaces for
giving
services to
the outside
world
(north/south
traffic).</span></p>
<p><br>
<span></span></p>
<p>This said, my
first question
is: How should
I deploy
GlusterFS in
such oVirt
scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I
create 3 RAID
(i.e.: RAID
5), one on
each oVirt
node, and then
create a
GlusterFS
using them?</p>
<p>2 - Instead,
should I
create a JBOD
array made of
all server's
disks?</p>
<p>3 - What is
the best
Gluster
configuration
to provide for
HA while not
consuming too
much disk
space?<br>
</p>
<p>4 - Does a
oVirt
hypervisor pod
like I am
planning to
build, and the
virtualization
environment,
benefits from
tiering when
using a SSD
disk? And yes,
will Gluster
do it by
default or I
have to
configure it
to do so?</p>
<p><br>
</p>
<p>At the bottom
line, what is
the good
practice for
using
GlusterFS in
small pods for
enterprises?<br>
</p>
<p><br>
</p>
<p>You
opinion/feedback
will be really
appreciated!</p>
<span
class="m_1874859859842763104m_4831886883305672718HOEnZb"><font
color="#888888">
<p>Moacir<br>
</p>
</font></span></div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a
href="mailto:Users@ovirt.org"
target="_blank"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr...
<br>
</blockquote>
</div>
<br>
</div>
</span></div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/<wbr>mai...
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------A95CF1C0E4312049A540BB6A--
9:48 p.m.
On Mon, Aug 7, 2017 at 2:41 PM, Colin Coe <colin.coe(a)gmail.com> wrote:
Hi
I just thought that you'd do hardware RAID if you had the controller or
JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pretty
likely to have a hardware RAID controller. I've never done JBOD with
hardware RAID. I think having a single gluster brick on hardware JBOD
would be riskier than multiple bricks, each on a single disk, but thats not
based on anything other than my prejudices.
I thought gluster tiering was for the most frequently accessed files, in
which case all the VMs disks would end up in the hot tier. However, I have
been wrong before...
The most frequent shards, may not be complete files.
Y.
I just wanted to know where the OS was going as I didn't see it
mentioned
in the OP. Normally, I'd have the OS on a RAID1 but in your case thats a
lot of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and made the
important point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <
moacirferreira(a)hotmail.com> wrote:
> Hi Colin,
>
>
> I am in Portugal, so sorry for this late response. It is quite confusing
> for me, please consider:
>
>
> 1* - *What if the RAID is done by the server's disk controller, not by
> software?
>
> 2 - For JBOD I am just using gdeploy to deploy it. However, I am not
> using the oVirt node GUI to do this.
>
>
> 3 - As the VM .qcow2 files are quite big, tiering would only help if
> made by an intelligent system that uses SSD for chunks of data not for the
> entire .qcow2 file. But I guess this is a problem everybody else has. So,
> Do you know how tiering works in Gluster?
>
>
> 4 - I am putting the OS on the first disk. However, would you do
> differently?
>
>
> Moacir
>
> ------------------------------
> *From:* Colin Coe <colin.coe(a)gmail.com>
> *Sent:* Monday, August 7, 2017 4:48 AM
> *To:* Moacir Ferreira
> *Cc:* users(a)ovirt.org
> *Subject:* Re: [ovirt-users] Good practices
>
> 1) RAID5 may be a performance hit-
>
> 2) I'd be inclined to do this as JBOD by creating a distributed disperse
> volume on each server. Something like
>
> echo gluster volume create dispersevol disperse-data 5 redundancy 2 \
> $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e
> "server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done)
>
> 3) I think the above.
>
> 4) Gluster does support tiering, but IIRC you'd need the same number of
> SSD as spindle drives. There may be another way to use the SSD as a fast
> cache.
>
> Where are you putting the OS?
>
> Hope I understood the question...
>
> Thanks
>
> On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <
> moacirferreira(a)hotmail.com> wrote:
>
>> I am willing to assemble a oVirt "pod", made of 3 servers, each with 2
>> CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
>> GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
>> a dual 10Gb NIC. So my intention is to create a loop like a server triangle
>> using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
>> move VMs around the pod (east /west traffic) while using the 10Gb
>> interfaces for giving services to the outside world (north/south traffic).
>>
>>
>> This said, my first question is: How should I deploy GlusterFS in such
>> oVirt scenario? My questions are:
>>
>>
>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
>> then create a GlusterFS using them?
>>
>> 2 - Instead, should I create a JBOD array made of all server's disks?
>>
>> 3 - What is the best Gluster configuration to provide for HA while not
>> consuming too much disk space?
>>
>> 4 - Does a oVirt hypervisor pod like I am planning to build, and the
>> virtualization environment, benefits from tiering when using a SSD disk?
>> And yes, will Gluster do it by default or I have to configure it to do so?
>>
>>
>> At the bottom line, what is the good practice for using GlusterFS in
>> small pods for enterprises?
>>
>>
>> You opinion/feedback will be really appreciated!
>>
>> Moacir
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:42 a.m.
On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.com>
wrote:
I am willing to assemble a oVirt "pod", made of 3 servers,
each with 2 CPU
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
a dual 10Gb NIC. So my intention is to create a loop like a server triangle
using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
move VMs around the pod (east /west traffic) while using the 10Gb
interfaces for giving services to the outside world (north/south traffic).
Very nice gear. How are you planning the network exactly? Without a switch,
back-to-back? (sounds OK to me, just wanted to ensure this is what the
'dual' is used for). However, I'm unsure if you have the correct balance
between the interface speeds (40g) and the disks (too many HDDs?).
This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
then create a GlusterFS using them?
I would assume RAID 1 for the operating system (you don't want a single
point of failure there?) and the rest JBODs. The SSD will be used for
caching, I reckon? (I personally would add more SSDs instead of HDDs, but
it does depend on the disk sizes and your space requirements.
2 - Instead, should I create a JBOD array made of all server's
disks?
3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?
Replica 2 + Arbiter sounds good to me.
4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so?
Yes, I believe using lvmcache is the best way to go.
At the bottom line, what is the good practice for using GlusterFS in small
pods for enterprises?
Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5). Sharding
(enabled out of the box if you use a hyper-converged setup via gdeploy).
Y.
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
1:13 p.m.
--_000_VI1P190MB02856C6CA513853E8EDA2DB2C8B50VI1P190MB0285EURP_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi, in-line responses.
Thanks,
Moacir
________________________________
From: Yaniv Kaul <ykaul(a)redhat.com Sent:
Monday, August 7, 2017 7:42 AM
To: Moacir Ferreira
Cc: users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira <moacirferreira(a)hotmail.com=
<mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
Very nice gear. How are you planning the network exactly? Without a switch,=
back-to-back? (sounds OK to me, just wanted to ensure this is what the 'du=
al' is used for). However, I'm unsure if you have the correct balance betwe=
en the interface speeds (40g) and the disks (too many HDDs?).
Moacir: The idea is to have a very high performance network for the distrib=
uted file system and to prevent bottlenecks when we move one VM from a node=
to another. Using 40Gb NICs I can just connect the servers back-to-back. I=
n this case I don't need the expensive 40Gb switch, I get very high speed a=
nd no contention between north/south traffic with east/west.
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
I would assume RAID 1 for the operating system (you don't want a single poi=
nt of failure there?) and the rest JBODs. The SSD will be used for caching,=
I reckon? (I personally would add more SSDs instead of HDDs, but it does d=
epend on the disk sizes and your space requirements.
Moacir: Yes, I agree that I need a RAID-1 for the OS. Now, generic JBOD or =
a JBOD assembled using RAID-5 "disks" created by the server's disk
controll=
er?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
Replica 2 + Arbiter sounds good to me.
Moacir: I agree, and that is what I am using.
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
Yes, I believe using lvmcache is the best way to go.
Moacir: Are you sure? I say that because the qcow2 files will be quite big.=
So if tiering is "file based" the SSD would have to be very, very big unle=
ss Gluster tiering do it by "chunks of data".
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5). Sharding (=
enabled out of the box if you use a hyper-converged setup via gdeploy).
Moacir: Yes! This is another reason to have separate networks for north/sou=
th and east/west. In that way I can use the standard MTU on the 10Gb NICs a=
nd jumbo frames on the file/move 40Gb NICs.
Y.
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_VI1P190MB02856C6CA513853E8EDA2DB2C8B50VI1P190MB0285EURP_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Hi, in-line responses.<br </p <br
<p>Thanks,</p <p>Moacir<br </p <br <div
style=3D"color: rgb(49, 55, 57);" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Yaniv Kaul <ykau=
l(a)redhat.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 7:42 AM<br <b>To:</b> Moacir
Ferreira<br <b>Cc:</b>
users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
dir=3D"ltr"><br <div
class=3D"gmail_extra"><br <div
class=3D"gmail_quote">On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira =
<span dir=3D"ltr" <<a
href=3D"mailto:moacirferreira@hotmail.com"
target=3D"_blank">moacirf=
erreira(a)hotmail.com</a>&gt;</span> wrote:<br <blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_5509585889569690791divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p><span>I am willing to assemble a oVirt
"pod", made of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
</div </div </blockquote
<div><br </div <div>Very nice gear. How are you planning the network
exactly? Without a sw=
itch, back-to-back? (sounds OK to me, just wanted to ensure this is what th=
e 'dual' is used for). However, I'm unsure if you have the correct balance =
between the interface speeds (40g)
and the disks (too many HDDs?).</div
<div><br <span style=3D"color: rgb(0,
0, 0);">M</span><span style=3D"color: rgb(0, 0=
, 0);">o</span><span style=3D"color: rgb(0, 0,
0);">a</span><span style=3D"=
color: rgb(0, 0, 0);">c</span><span style=3D"color: rgb(0, 0,
0);">i</span>=
<span style=3D"color: rgb(0, 0, 0);">r</span><span
style=3D"color: rgb(0, 0=
, 0);">:</span><span style=3D"color: rgb(0, 0, 0);" The idea is to have a very high performance network for the
distributed fi=
le system and to prevent bottlenecks when we move one VM from a node to ano=
ther. Using
</span><span style=3D"color: rgb(0, 0, 0);">40Gb NICs I can just
connect th=
e servers back-to-back. In this case I don't need the expensive 40Gb switch=
, I get very high speed and no contention between north/south traffic with =
east/west.</span><br <br
</div <blockquote
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_5509585889569690791divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p><br
<span></span></p
<p>This said, my first question is: How should I deploy GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
</div </div </blockquote
<div>I would assume RAID 1 for the operating system (you don't want a
singl=
e point of failure there?) and the rest JBODs. The SSD will be used for cac=
hing, I reckon? (I personally would add more SSDs instead of HDDs, but it d=
oes depend on the disk sizes and
your space requirements.<br <br <span
style=3D"color: rgb(0, 0, 0);">Moacir: Yes, I agree that I need a RAI=
D-1 for the OS. Now, generic JBOD or a JBOD assembled using RAID-5 "di=
sks" created</span><span style=3D"color: rgb(0, 0, 0);"> by
the server=
's disk
</span><span style=3D"color: rgb(0, 0,
0);">controller?</span><br
</div <span style=3D"color: rgb(0,
0, 0);"></span <div> <br </div
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;
border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_5509585889569690791divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p>2 - Instead, should I create a JBOD array made of all
server's disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p </div </div
</blockquote <div><br </div
<div>Replica 2 + Arbiter sounds good to me.</div <div><span style=3D"color: rgb(0, 0,
0);">M</span><span style=3D"color: rgb=
(0, 0, 0);">o</span><span style=3D"color: rgb(0, 0,
0);">a</span><span styl=
e=3D"color: rgb(0, 0, 0);">c</span><span style=3D"color: rgb(0,
0, 0);">i</=
span><span style=3D"color: rgb(0, 0, 0);">r</span><span
style=3D"color: rgb=
(0, 0, 0);">:</span><span style=3D"color: rgb(0, 0, 0);" </span><span style=3D"color: rgb(0, 0,
0);">I agree, and that is what I am =
using.</span><br <br
</div <blockquote
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_5509585889569690791divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p></p
<p>4 - Does a oVirt hypervisor pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p </div
</div </blockquote <div><br
</div <div>Yes, I believe using
lvmcache is the best way to go. </div
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;
border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_5509585889569690791divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p>Moacir: Are you sure? I say that because the qcow2
files will be quite b=
ig. So if tiering is "file based" the SSD would have to be very, =
very big unless Gluster tiering do it by "chunks of data".<br </p <p><br </p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
</div </div </blockquote
<div><br </div <div>Don't forget jumbo frames. libgfapi (coming
hopefully in 4.1.5). Shard=
ing (enabled out of the box if you use a hyper-converged setup via gdeploy)=
.<br <b>Moacir:</b> Yes! This is another reason to have
separate networks for no=
rth/south and east/west. In that way I can use the standard MTU on the 10Gb=
NICs and jumbo frames on the file/move 40Gb NICs.<br
<br </div <div>Y.</div
<div> </div
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;
border-left:1=
px #ccc solid; padding-left:1ex" <div
dir=3D"ltr" <div
id=3D"m_5509585889569690791divtagdefaultwrapper" dir=3D"ltr"
style=3D"=
font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif" <p></p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <span class=3D"HOEnZb"><font
color=3D"#888888" <p>Moacir<br </p
</font></span></div
</div <br
______________________________<wbr>_________________<br Users mailing list<br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer=
"
target=3D"_blank">http://lists.ovirt.org/<wbr>mailman/...
br <br </blockquote </div
<br </div </div
</div </div </div
</body </html
--_000_VI1P190MB02856C6CA513853E8EDA2DB2C8B50VI1P190MB0285EURP_--
4:08 p.m.
This is a multi-part message in MIME format.
--------------7999E232532992C2B4184FF7
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Moacir, I beleive for using the 3 servers directly connected to each
others without switch you have to have a Bridge on each server for every
2 physical interfaces to allow the traffic passthrough in layer2 (Is it
possible to create this from the oVirt Engine Web Interface?). If your
ovirtmgmt network is separate from other (should really be) that should
be fine to do.
Fernando
On 07/08/2017 07:13, Moacir Ferreira wrote:
Hi, in-line responses.
Thanks,
Moacir
------------------------------------------------------------------------
*From:* Yaniv Kaul <ykaul(a)redhat.com>
*Sent:* Monday, August 7, 2017 7:42 AM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org
*Subject:* Re: [ovirt-users] Good practices
On Sun, Aug 6, 2017 at 5:49 PM, Moacir Ferreira
<moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each
with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
idea is to use GlusterFS to provide HA for the VMs. The 3 servers
have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to
create a loop like a server triangle using the 40Gb NICs for
virtualization files (VMs .qcow2) access and to move VMs around
the pod (east /west traffic) while using the 10Gb interfaces for
giving services to the outside world (north/south traffic).
Very nice gear. How are you planning the network exactly? Without a
switch, back-to-back? (sounds OK to me, just wanted to ensure this is
what the 'dual' is used for). However, I'm unsure if you have the
correct balance between the interface speeds (40g) and the disks (too
many HDDs?).
Moacir:The idea is to have a very high performance network for the
distributed file system and to prevent bottlenecks when we move one VM
from a node to another. Using 40Gb NICs I can just connect the servers
back-to-back. In this case I don't need the expensive 40Gb switch, I
get very high speed and no contention between north/south traffic with
east/west.
This said, my first question is: How should I deploy GlusterFS in
such oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node,
and then create a GlusterFS using them?
I would assume RAID 1 for the operating system (you don't want a
single point of failure there?) and the rest JBODs. The SSD will be
used for caching, I reckon? (I personally would add more SSDs instead
of HDDs, but it does depend on the disk sizes and your space requirements.
Moacir: Yes, I agree that I need a RAID-1 for the OS. Now, generic
JBOD or a JBOD assembled using RAID-5 "disks" createdby the server's
disk controller?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while
not consuming too much disk space?
Replica 2 + Arbiter sounds good to me.
Moacir:I agree, and that is what I am using.
4 - Does a oVirt hypervisor pod like I am planning to build, and
the virtualization environment, benefits from tiering when using a
SSD disk? And yes, will Gluster do it by default or I have to
configure it to do so?
Yes, I believe using lvmcache is the best way to go.
Moacir: Are you sure? I say that because the qcow2 files will be
quite big. So if tiering is "file based" the SSD would have to be
very, very big unless Gluster tiering do it by "chunks of data".
At the bottom line, what is the good practice for using GlusterFS
in small pods for enterprises?
Don't forget jumbo frames. libgfapi (coming hopefully in 4.1.5).
Sharding (enabled out of the box if you use a hyper-converged setup
via gdeploy).
*Moacir:* Yes! This is another reason to have separate networks for
north/south and east/west. In that way I can use the standard MTU on
the 10Gb NICs and jumbo frames on the file/move 40Gb NICs.
Y.
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------7999E232532992C2B4184FF7
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Moacir, I beleive for using the 3 servers directly connected to
each others without switch you have to have a Bridge on each
server for every 2 physical interfaces to allow the traffic
passthrough in layer2 (Is it possible to create this from the
oVirt Engine Web Interface?). If your ovirtmgmt network is
separate from other (should really be) that should be fine to do.</p>
<p><br>
</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 07:13, Moacir Ferreira
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:VI1P190MB02856C6CA513853E8EDA2DB2C8B50@VI1P190MB0285.EURP190.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
dir="ltr">
<p>Hi, in-line responses.<br>
</p>
<br>
<p>Thanks,</p>
<p>Moacir<br>
</p>
<br>
<div style="color: rgb(49, 55, 57);">
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
Yaniv Kaul <a class="moz-txt-link-rfc2396E"
href="mailto:ykaul@redhat.com"><ykaul@redhat.com></a><br>
<b>Sent:</b> Monday, August 7, 2017 7:42 AM<br>
<b>To:</b> Moacir Ferreira<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated"
href="mailto:users@ovirt.org">users@ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] Good practices</font>
<div> </div>
</div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Aug 6, 2017 at 5:49 PM,
Moacir Ferreira <span dir="ltr">
<<a href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div
id="m_5509585889569690791divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p><span>I am willing to assemble a oVirt
"pod",
made of 3 servers, each with 2 CPU sockets
of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD.
The idea is to use GlusterFS to provide HA
for the VMs. The 3 servers have a dual 40Gb
NIC and a dual 10Gb NIC. So my intention is
to create a loop like a server triangle
using the 40Gb NICs for virtualization files
(VMs .qcow2) access and to move VMs around
the pod (east /west traffic) while using the
10Gb interfaces for giving services to the
outside world (north/south traffic).</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Very nice gear. How are you planning the network
exactly? Without a switch, back-to-back? (sounds OK
to me, just wanted to ensure this is what the 'dual'
is used for). However, I'm unsure if you have the
correct balance between the interface speeds (40g)
and the disks (too many HDDs?).</div>
<div><br>
<span style="color: rgb(0, 0,
0);">M</span><span
style="color: rgb(0, 0, 0);">o</span><span
style="color: rgb(0, 0, 0);">a</span><span
style="color: rgb(0, 0, 0);">c</span><span
style="color: rgb(0, 0, 0);">i</span><span
style="color: rgb(0, 0, 0);">r</span><span
style="color: rgb(0, 0, 0);">:</span><span
style="color: rgb(0, 0, 0);"> The idea is to have
a very high performance network for the
distributed file system and to prevent bottlenecks
when we move one VM from a node to another. Using
</span><span style="color: rgb(0, 0, 0);">40Gb
NICs
I can just connect the servers back-to-back. In
this case I don't need the expensive 40Gb switch,
I get very high speed and no contention between
north/south traffic with east/west.</span><br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div
id="m_5509585889569690791divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p><br>
<span></span></p>
<p>This said, my first question is: How should I
deploy GlusterFS in such oVirt scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5),
one on each oVirt node, and then create a
GlusterFS using them?</p>
</div>
</div>
</blockquote>
<div>I would assume RAID 1 for the operating system
(you don't want a single point of failure there?)
and the rest JBODs. The SSD will be used for
caching, I reckon? (I personally would add more SSDs
instead of HDDs, but it does depend on the disk
sizes and your space requirements.<br>
<br>
<span style="color: rgb(0, 0, 0);">Moacir: Yes, I
agree that I need a RAID-1 for the OS. Now,
generic JBOD or a JBOD assembled using RAID-5
"disks" created</span><span style="color:
rgb(0,
0, 0);"> by the server's disk
</span><span style="color: rgb(0, 0,
0);">controller?</span><br>
</div>
<span style="color: rgb(0, 0, 0);"></span>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div
id="m_5509585889569690791divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>2 - Instead, should I create a JBOD array
made of all server's disks?</p>
<p>3 - What is the best Gluster configuration to
provide for HA while not consuming too much
disk space?<br>
</p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Replica 2 + Arbiter sounds good to me.</div>
<div><span style="color: rgb(0, 0,
0);">M</span><span
style="color: rgb(0, 0, 0);">o</span><span
style="color: rgb(0, 0, 0);">a</span><span
style="color: rgb(0, 0, 0);">c</span><span
style="color: rgb(0, 0, 0);">i</span><span
style="color: rgb(0, 0, 0);">r</span><span
style="color: rgb(0, 0, 0);">:</span><span
style="color: rgb(0, 0, 0);">
</span><span style="color: rgb(0, 0, 0);">I
agree,
and that is what I am using.</span><br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div
id="m_5509585889569690791divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>4 - Does a oVirt hypervisor pod like I am
planning to build, and the virtualization
environment, benefits from tiering when using
a SSD disk? And yes, will Gluster do it by
default or I have to configure it to do so?</p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, I believe using lvmcache is the best way to
go. </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div
id="m_5509585889569690791divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>Moacir: Are you sure? I say that because the
qcow2 files will be quite big. So if tiering
is "file based" the SSD would have to be very,
very big unless Gluster tiering do it by
"chunks of data".<br>
</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice
for using GlusterFS in small pods for
enterprises?<br>
</p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Don't forget jumbo frames. libgfapi (coming
hopefully in 4.1.5). Sharding (enabled out of the
box if you use a hyper-converged setup via gdeploy).<br>
<b>Moacir:</b> Yes! This is another reason to have
separate networks for north/south and east/west. In
that way I can use the standard MTU on the 10Gb NICs
and jumbo frames on the file/move 40Gb NICs.<br>
<br>
</div>
<div>Y.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div
id="m_5509585889569690791divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p><br>
</p>
<p>You opinion/feedback will be really
appreciated!</p>
<span class="HOEnZb"><font
color="#888888">
<p>Moacir<br>
</p>
</font></span></div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/<wbr>mai...
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------7999E232532992C2B4184FF7--
10:50 p.m.
--Apple-Mail=_E33F2A30-713D-4209-8D44-2134E4046305
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
> Moacir: Yes! This is another reason to have separate networks for
=
north/south and east/west. In that way I can use the standard MTU on the =
10Gb NICs and jumbo frames on the file/move 40Gb NICs.
Why not Jumbo frame every where ?=
--Apple-Mail=_E33F2A30-713D-4209-8D44-2134E4046305
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word;
=
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div><blockquote type=3D"cite"
class=3D""><div =
class=3D""><blockquote type=3D"cite" =
cite=3D"mid:VI1P190MB02856C6CA513853E8EDA2DB2C8B50@VI1P190MB0285.EURP190.P=
ROD.OUTLOOK.COM" style=3D"font-family: Helvetica; font-size: 12px; =
font-style: normal; font-variant-caps: normal; font-weight: normal; =
letter-spacing: normal; orphans: auto; text-align: start; text-indent: =
0px; text-transform: none; white-space: normal; widows: auto; =
word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: =
rgb(255, 255, 255);" class=3D""><div
id=3D"divtagdefaultwrapper" =
dir=3D"ltr" style=3D"font-size: 12pt; font-family: Calibri, Helvetica, =
sans-serif;" class=3D""><div style=3D"color: rgb(49, 55,
57);" =
class=3D""><div class=3D""><div dir=3D"ltr"
class=3D""><div =
class=3D"gmail_extra"><div class=3D"gmail_quote"><div
class=3D""><b =
class=3D"">Moacir:</b><span =
class=3D"Apple-converted-space"> </span>Yes! This is another
reason =
to have separate networks for north/south and east/west. In that way I =
can use the standard MTU on the 10Gb NICs and jumbo frames on the =
file/move 40Gb NICs.<br =
class=3D""></div></div></div></div></div></div></div></blockquote></div></=
blockquote></div><br class=3D""><div
class=3D"">Why not Jumbo frame =
every where ?</div></body></html>=
--Apple-Mail=_E33F2A30-713D-4209-8D44-2134E4046305--
9:46 a.m.
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for several
different companies, and have dealt with the Red Hat Support Team in depth
about optimal configuration in regards to setting up GlusterFS most
efficiently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the
system as just a JBOD, sure there is some protection by having the data
replicated, however, the recommendation is to use RAID 6 (preferred) or
RAID-5, or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
*"A typical Gluster configuration would use RAID underneath the bricks.
RAID 6 is most typical as it gives you 2 disk failure protection, but RAID
5 could be used too. Once you have the RAIDed bricks, you'd then apply the
desired replication on top of that. The most popular way of doing this
would be distributed replicated with 2x replication. In general you'll get
better performance with larger bricks. 12 drives is often a sweet spot.
Another option would be to create a separate tier using all SSD’s.” *
*In order to SSD tiering from my understanding you would need 1 x NVMe
drive in each server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME). So with you only having 1
SSD drive in each server, I’d suggest maybe looking into the NVME option. *
*Since your using only 3-servers, what I’d probably suggest is to do (2
Replicas + Arbiter Node), this setup actually doesn’t require the 3rd
server to have big drives at all as it only stores meta-data about the
files and not actually a full copy. *
*Please see the attached document that was given to me by Red Hat to get
more information on this. Hope this information helps you.*
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.com)
wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
a dual 10Gb NIC. So my intention is to create a loop like a server triangle
using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
move VMs around the pod (east /west traffic) while using the 10Gb
interfaces for giving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
1:19 p.m.
--_000_VI1P190MB0285197D4F878904201462F9C8B50VI1P190MB0285EURP_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Devin,
Many, many thaks for your response. I will read the doc you sent and if I s=
till have questions I will post them here.
But why would I use a RAIDed brick if Gluster, by itself, already "protects=
" the data by making replicas. You see, that is what is confusing to me...
Thanks,
Moacir
________________________________
From: Devin Acosta <devin(a)pabstatencio.com Sent:
Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for several=
different companies, and have dealt with the Red Hat Support Team in depth=
about optimal configuration in regards to setting up GlusterFS most effici=
ently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the s=
ystem as just a JBOD, sure there is some protection by having the data repl=
icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,=
or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
"A typical Gluster configuration would use RAID underneath the bricks. RAID=
6 is most typical as it gives you 2 disk failure protection, but RAID 5 co=
uld be used too. Once you have the RAIDed bricks, you'd then apply the desi=
red replication on top of that. The most popular way of doing this would be=
distributed replicated with 2x replication. In general you'll get better p=
erformance with larger bricks. 12 drives is often a sweet spot. Another opt=
ion would be to create a separate tier using all SSD=92s.=94
In order to SSD tiering from my understanding you would need 1 x NVMe drive=
in each server, or 4 x SSD hot tier (it needs to be distributed, replicate=
d for the hot tier if not using NVME). So with you only having 1 SSD drive =
in each server, I=92d suggest maybe looking into the NVME option.
Since your using only 3-servers, what I=92d probably suggest is to do (2 Re=
plicas + Arbiter Node), this setup actually doesn=92t require the 3rd serve=
r to have big drives at all as it only stores meta-data about the files and=
not actually a full copy.
Please see the attached document that was given to me by Red Hat to get mor=
e information on this. Hope this information helps you.
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_VI1P190MB0285197D4F878904201462F9C8B50VI1P190MB0285EURP_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3DWindows-1=
252" <style
type=3D"text/css" style=3D"display:none;"><!-- P
{margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Devin,</p
<p><br </p
<p>Many, many thaks for your response. I will read the doc you sent and if =
I still have questions I will post them here.</p
<p><br </p
<p>But why would I use a RAIDed brick if Gluster, by itself, already
"=
protects" the data by making replicas. You see, that is what is confus=
ing to me...
<br </p <br
<p>Thanks,</p <p>Moacir<br </p <br
<br <div style=3D"color: rgb(49,
55, 57);" <hr tabindex=3D"-1"
style=3D"display:inline-block; width:98%" <div
id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Devin Acosta <de=
vin(a)pabstatencio.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 7:46 AM<br <b>To:</b> Moacir
Ferreira; users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">Moacir,</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">I have recently installed multiple Red Hat Virtualizatio=
n hosts for several different companies, and have dealt with the Red Hat Su=
pport Team in depth about optimal configuration
in regards to setting up GlusterFS most efficiently and I wanted to share =
with you what I learned.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">In general Red Hat Virtualization team frowns upon using=
each DISK of the system as just a JBOD, sure there is some protection by h=
aving the data replicated, however, the
recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 =
at the very least.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><br </div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<br <div
id=3D"bloop_sign_1502087376725469184" class=3D"bloop_sign"><span
style=
=3D"font-family:'helvetica Neue',helvetica;
font-size:14px">--</span><br st=
yle=3D"font-family:'helvetica Neue',helvetica; font-size:14px" <div class=3D"gmail_signature"
style=3D"font-family:'helvetica Neue',helvet=
ica; font-size:14px" <div dir=3D"ltr" <div><br
</div <div>Devin Acosta, RHCA,
RHVCA</div <div>Red Hat Certified
Architect</div <div></div </div
</div </div <br <p
class=3D"airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (<=
a
href=3D"mailto:moacirferreira@hotmail.com">moacirferreira@hotmail.com</a>=
) wrote:</p <blockquote
type=3D"cite" class=3D"clean_bq"><span <div dir=3D"ltr"
<div></div <div <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p><span>I am willing to assemble a oVirt "pod", made
of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br </p
<p>This said, my first question is: How should I deploy GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <p>Moacir<br
</p </div _______________________________________________ <br Users mailing list <br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a> <br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
t.org/mailman/listinfo/users</a <br
</div </div </span></blockquote
</div </div </div
</body </html
--_000_VI1P190MB0285197D4F878904201462F9C8B50VI1P190MB0285EURP_--
Here is the direct quote from Red Hat when I asked about RAID
and Bri=
cks:</font></div<i><br>
</i></font></div<i>"A typical Gluster configuration would use RAID
underneath the bri=
cks. RAID 6 is most typical as it gives you 2 disk failure
protection, but =
RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired replication on top of that. Th=
e most popular way of doing this would be distributed replicated with 2x re=
plication. In general you'll get better performance with larger bricks=
. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all SSD=92s.=94 </i><=
/font></div<i>In order to SSD tiering from my understanding you
would need 1 x N=
VMe drive in each server, or 4 x SSD hot tier (it needs to be
distributed, =
replicated for the hot tier if not using NVME).
So with you only having 1 SSD drive in each server, I=92d suggest may=
be looking into the NVME option. </i></font></div<i><br>
</i></font></div<i>Since your using only 3-servers, what I=92d probably suggest
is to do (=
2 Replicas + Arbiter Node), this setup actually doesn=92t
require the 3=
rd server to have big drives at all as it only
stores meta-data about the files and not actually a full copy. </i></=
font></div<i><br>
</i></font></div<i>Please see the attached document that was given to me by Red
Hat to get=
more information on this. Hope this information helps
you.</i></font></div=
<i><br>
</i></font></div
4:04 p.m.
This is a multi-part message in MIME format.
--------------ECDF126785BB7A19D9FCBAF5
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
For any RAID 5 or 6 configuration I normally follow a simple gold rule
which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6
However I didn't really understand well the recommendation to use any
RAID with GlusterFS. I always thought that GlusteFS likes to work in
JBOD mode and control the disks (bricks) directlly so you can create
whatever distribution rule you wish, and if a single disk fails you just
replace it and which obviously have the data replicated from another.
The only downside of using in this way is that the replication data will
be flow accross all servers but that is not much a big issue.
Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
Thanks
Regards
Fernando
On 07/08/2017 03:46, Devin Acosta wrote:
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for
several different companies, and have dealt with the Red Hat Support
Team in depth about optimal configuration in regards to setting up
GlusterFS most efficiently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of
the system as just a JBOD, sure there is some protection by having the
data replicated, however, the recommendation is to use RAID 6
(preferred) or RAID-5, or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
/
/
/"A typical Gluster configuration would use RAID underneath the
bricks. RAID 6 is most typical as it gives you 2 disk failure
protection, but RAID 5 could be used too. Once you have the RAIDed
bricks, you'd then apply the desired replication on top of that. The
most popular way of doing this would be distributed replicated with 2x
replication. In general you'll get better performance with larger
bricks. 12 drives is often a sweet spot. Another option would be to
create a separate tier using all SSD’s.” /
/In order to SSD tiering from my understanding you would need 1 x NVMe
drive in each server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME). So with you only
having 1 SSD drive in each server, I’d suggest maybe looking into the
NVME option. /
/
/
/Since your using only 3-servers, what I’d probably suggest is to do
(2 Replicas + Arbiter Node), this setup actually doesn’t require the
3rd server to have big drives at all as it only stores meta-data about
the files and not actually a full copy. /
/
/
/Please see the attached document that was given to me by Red Hat to
get more information on this. Hope this information helps you./
/
/
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
(moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>) wrote:
> I am willing to assemble a oVirt "pod", made of 3 servers, each with
> 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is
> to use GlusterFS to provide HA for the VMs. The 3 servers have a dual
> 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop
> like a server triangle using the 40Gb NICs for virtualization files
> (VMs .qcow2) access and to move VMs around the pod (east /west
> traffic) while using the 10Gb interfaces for giving services to the
> outside world (north/south traffic).
>
>
> This said, my first question is: How should I deploy GlusterFS in
> such oVirt scenario? My questions are:
>
>
> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node,
> and then create a GlusterFS using them?
>
> 2 - Instead, should I create a JBOD array made of all server's disks?
>
> 3 - What is the best Gluster configuration to provide for HA while
> not consuming too much disk space?
>
> 4 - Does a oVirt hypervisor pod like I am planning to build, and the
> virtualization environment, benefits from tiering when using a SSD
> disk? And yes, will Gluster do it by default or I have to configure
> it to do so?
>
>
> At the bottom line, what is the good practice for using GlusterFS in
> small pods for enterprises?
>
>
> You opinion/feedback will be really appreciated!
>
> Moacir
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------ECDF126785BB7A19D9FCBAF5
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>For any RAID 5 or 6 configuration I normally follow a simple gold
rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the recommendation to use
any RAID with GlusterFS. I always thought that GlusteFS likes to
work in JBOD mode and control the disks (bricks) directlly so you
can create whatever distribution rule you wish, and if a single
disk fails you just replace it and which obviously have the data
replicated from another. The only downside of using in this way is
that the replication data will be flow accross all servers but
that is not much a big issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and JBOD +
GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin Acosta
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">I have recently installed multiple Red Hat
Virtualization hosts for several different companies, and have
dealt with the Red Hat Support Team in depth about optimal
configuration in regards to setting up GlusterFS most
efficiently and I wanted to share with you what I
learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">In general Red Hat Virtualization team
frowns upon using each DISK of the system as just a JBOD, sure
there is some protection by having the data replicated,
however, the recommendation is to use RAID 6 (preferred) or
RAID-5, or at least RAID-1 at the very least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono">Here is the direct quote from Red Hat when I asked about
RAID and Bricks:</font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>"A typical Gluster configuration would use RAID
underneath the bricks. RAID 6 is most typical as it gives
you 2 disk failure protection, but RAID 5 could be used
too. Once you have the RAIDed bricks, you'd then apply the
desired replication on top of that. The most popular way of
doing this would be distributed replicated with 2x
replication. In general you'll get better performance with
larger bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all
SSD’s.” </i></font></div>
<div id="bloop_customfont" style="margin:0px"><br>
</div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>In order to SSD tiering from my understanding you
would need 1 x NVMe drive in each server, or 4 x SSD hot
tier (it needs to be distributed, replicated for the hot
tier if not using NVME). So with you only having 1 SSD drive
in each server, I’d suggest maybe looking into the NVME
option. </i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>Since your using only 3-servers, what I’d probably
suggest is to do (2 Replicas + Arbiter Node), this setup
actually doesn’t require the 3rd server to have big drives
at all as it only stores meta-data about the files and not
actually a full copy. </i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>Please see the attached document that was given to me
by Red Hat to get more information on this. Hope this
information helps you.</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature" style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir
Ferreira (<a href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite" class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt "pod",
made of
3 servers,
each with 2 CPU sockets of 12 cores, 256GB RAM, 7
HDD 10K, 1 SSD.
The idea is to use GlusterFS to provide HA for the
VMs. The 3
servers have a dual 40Gb NIC and a dual 10Gb NIC. So
my intention
is to create a loop like a server triangle using the
40Gb NICs for
virtualization files (VMs .qcow2) access and to move
VMs around the
pod (east /west traffic) while using the 10Gb
interfaces for giving
services to the outside world (north/south
traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I deploy
GlusterFS
in such oVirt scenario? My questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on
each oVirt
node, and then create a GlusterFS using them?</p>
<p>2 - Instead, should I create a JBOD array made of all
server's
disks?</p>
<p>3 - What is the best Gluster configuration to provide
for HA
while not consuming too much disk space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am planning to
build, and
the virtualization environment, benefits from tiering
when using a
SSD disk? And yes, will Gluster do it by default or I
have to
configure it to do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice for
using
GlusterFS in small pods for enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________
<br>
Users mailing list
<br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------ECDF126785BB7A19D9FCBAF5--
4:26 p.m.
This is a multi-part message in MIME format.
--------------C2F37362CE639F36E44EBBD8
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi Frenando,
Here is my experience, if you consider a particular hard drive as a
brick for gluster volume and it dies, i.e. it becomes not accessible
it's a huge hassle to discard that brick and exchange with another one,
since gluster some tries to access that broken brick and it's causing
(at least it cause for me) a big pain, therefore it's better to have a
RAID as brick, i.e. have RAID 1 (mirroring) for each brick, in this case
if the disk is down you can easily exchange it and rebuild the RAID
without going offline, i.e switching off the volume doing brick
manipulations and switching it back on.
Cheers
Erekle
On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
For any RAID 5 or 6 configuration I normally follow a simple gold rule
which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6
However I didn't really understand well the recommendation to use any
RAID with GlusterFS. I always thought that GlusteFS likes to work in
JBOD mode and control the disks (bricks) directlly so you can create
whatever distribution rule you wish, and if a single disk fails you
just replace it and which obviously have the data replicated from
another. The only downside of using in this way is that the
replication data will be flow accross all servers but that is not much
a big issue.
Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
Thanks
Regards
Fernando
On 07/08/2017 03:46, Devin Acosta wrote:
>
> Moacir,
>
> I have recently installed multiple Red Hat Virtualization hosts for
> several different companies, and have dealt with the Red Hat Support
> Team in depth about optimal configuration in regards to setting up
> GlusterFS most efficiently and I wanted to share with you what I learned.
>
> In general Red Hat Virtualization team frowns upon using each DISK of
> the system as just a JBOD, sure there is some protection by having
> the data replicated, however, the recommendation is to use RAID 6
> (preferred) or RAID-5, or at least RAID-1 at the very least.
>
> Here is the direct quote from Red Hat when I asked about RAID and Bricks:
> /
> /
> /"A typical Gluster configuration would use RAID underneath the
> bricks. RAID 6 is most typical as it gives you 2 disk failure
> protection, but RAID 5 could be used too. Once you have the RAIDed
> bricks, you'd then apply the desired replication on top of that. The
> most popular way of doing this would be distributed replicated with
> 2x replication. In general you'll get better performance with larger
> bricks. 12 drives is often a sweet spot. Another option would be to
> create a separate tier using all SSD’s.” /
>
> /In order to SSD tiering from my understanding you would need 1 x
> NVMe drive in each server, or 4 x SSD hot tier (it needs to be
> distributed, replicated for the hot tier if not using NVME). So with
> you only having 1 SSD drive in each server, I’d suggest maybe looking
> into the NVME option. /
> /
> /
> /Since your using only 3-servers, what I’d probably suggest is to do
> (2 Replicas + Arbiter Node), this setup actually doesn’t require the
> 3rd server to have big drives at all as it only stores meta-data
> about the files and not actually a full copy. /
> /
> /
> /Please see the attached document that was given to me by Red Hat to
> get more information on this. Hope this information helps you./
> /
> /
>
> --
>
> Devin Acosta, RHCA, RHVCA
> Red Hat Certified Architect
>
> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
> (moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>) wrote:
>
>> I am willing to assemble a oVirt "pod", made of 3 servers, each with
>> 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is
>> to use GlusterFS to provide HA for the VMs. The 3 servers have a
>> dual 40Gb NIC and a dual 10Gb NIC. So my intention is to create a
>> loop like a server triangle using the 40Gb NICs for virtualization
>> files (VMs .qcow2) access and to move VMs around the pod (east /west
>> traffic) while using the 10Gb interfaces for giving services to the
>> outside world (north/south traffic).
>>
>>
>> This said, my first question is: How should I deploy GlusterFS in
>> such oVirt scenario? My questions are:
>>
>>
>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node,
>> and then create a GlusterFS using them?
>>
>> 2 - Instead, should I create a JBOD array made of all server's disks?
>>
>> 3 - What is the best Gluster configuration to provide for HA while
>> not consuming too much disk space?
>>
>> 4 - Does a oVirt hypervisor pod like I am planning to build, and the
>> virtualization environment, benefits from tiering when using a SSD
>> disk? And yes, will Gluster do it by default or I have to configure
>> it to do so?
>>
>>
>> At the bottom line, what is the good practice for using GlusterFS in
>> small pods for enterprises?
>>
>>
>> You opinion/feedback will be really appreciated!
>>
>> Moacir
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------C2F37362CE639F36E44EBBD8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Frenando,</p>
<p>Here is my experience, if you consider a particular hard drive as
a brick for gluster volume and it dies, i.e. it becomes not
accessible it's a huge hassle to discard that brick and exchange
with another one, since gluster some tries to access that broken
brick and it's causing (at least it cause for me) a big pain,
therefore it's better to have a RAID as brick, i.e. have RAID 1
(mirroring) for each brick, in this case if the disk is down you
can easily exchange it and rebuild the RAID without going offline,
i.e switching off the volume doing brick manipulations and
switching it back on.<br>
</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 03:04 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>For any RAID 5 or 6 configuration I normally follow a simple
gold rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the recommendation to
use any RAID with GlusterFS. I always thought that GlusteFS
likes to work in JBOD mode and control the disks (bricks)
directlly so you can create whatever distribution rule you wish,
and if a single disk fails you just replace it and which
obviously have the data replicated from another. The only
downside of using in this way is that the replication data will
be flow accross all servers but that is not much a big issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and JBOD +
GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin Acosta
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">I have recently installed multiple Red Hat
Virtualization hosts for several different companies, and
have dealt with the Red Hat Support Team in depth about
optimal configuration in regards to setting up GlusterFS
most efficiently and I wanted to share with you what I
learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">In general Red Hat Virtualization team
frowns upon using each DISK of the system as just a JBOD,
sure there is some protection by having the data replicated,
however, the recommendation is to use RAID 6 (preferred) or
RAID-5, or at least RAID-1 at the very least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono">Here is the direct quote from Red Hat when I asked
about RAID and Bricks:</font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>"A typical Gluster configuration would use RAID
underneath the bricks. RAID 6 is most typical as it gives
you 2 disk failure protection, but RAID 5 could be used
too. Once you have the RAIDed bricks, you'd then apply the
desired replication on top of that. The most popular way
of doing this would be distributed replicated with 2x
replication. In general you'll get better performance with
larger bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all
SSD’s.” </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><br>
</div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>In order to SSD tiering from my understanding you
would need 1 x NVMe drive in each server, or 4 x SSD hot
tier (it needs to be distributed, replicated for the hot
tier if not using NVME). So with you only having 1 SSD
drive in each server, I’d suggest maybe looking into the
NVME option. </i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>Since your using only 3-servers, what I’d probably
suggest is to do (2 Replicas + Arbiter Node), this setup
actually doesn’t require the 3rd server to have big drives
at all as it only stores meta-data about the files and not
actually a full copy. </i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i>Please see the attached document that was given to
me by Red Hat to get more information on this. Hope this
information helps you.</i></font></div>
<div id="bloop_customfont" style="margin:0px"><font
face="Input
Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature"
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir
Ferreira (<a href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite" class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt "pod",
made
of 3 servers, each with 2 CPU sockets of 12 cores,
256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
GlusterFS to provide HA for the VMs. The 3 servers
have a dual 40Gb NIC and a dual 10Gb NIC. So my
intention is to create a loop like a server
triangle using the 40Gb NICs for virtualization
files (VMs .qcow2) access and to move VMs around
the pod (east /west traffic) while using the 10Gb
interfaces for giving services to the outside
world (north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I
deploy GlusterFS in such oVirt scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on
each oVirt node, and then create a GlusterFS using
them?</p>
<p>2 - Instead, should I create a JBOD array made of
all server's disks?</p>
<p>3 - What is the best Gluster configuration to
provide for HA while not consuming too much disk
space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am planning
to build, and the virtualization environment,
benefits from tiering when using a SSD disk? And
yes, will Gluster do it by default or I have to
configure it to do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice for
using GlusterFS in small pods for enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------C2F37362CE639F36E44EBBD8--
6:41 p.m.
This is a multi-part message in MIME format.
--------------A5088546CDF99006314CCEE8
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Thanks for the clarification Erekle.
However I get surprised with this way of operating from GlusterFS as it
adds another layer of complexity to the system (either a hardware or
software RAID) before the gluster config and increase the system's
overall costs.
An important point to consider is: In RAID configuration you already
have space 'wasted' in order to build redundancy (either RAID 1, 5, or
6). Then when you have GlusterFS on the top of several RAIDs you have
again more data replicated so you end up with the same data consuming
more space in a group of disks and again on the top of several RAIDs
depending on the Gluster configuration you have (in a RAID 1 config the
same data is replicated 4 times).
Yet another downside of having a RAID (specially RAID 5 or 6) is that it
reduces considerably the write speeds as each group of disks will end up
having the write speed of a single disk as all other disks of that group
have to wait for each other to write as well.
Therefore if Gluster already replicates data why does it create this big
pain you mentioned if the data is replicated somewhere else, can still
be retrieved to both serve clients and reconstruct the equivalent disk
when it is replaced ?
Fernando
On 07/08/2017 10:26, Erekle Magradze wrote:
Hi Frenando,
Here is my experience, if you consider a particular hard drive as a
brick for gluster volume and it dies, i.e. it becomes not accessible
it's a huge hassle to discard that brick and exchange with another
one, since gluster some tries to access that broken brick and it's
causing (at least it cause for me) a big pain, therefore it's better
to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick,
in this case if the disk is down you can easily exchange it and
rebuild the RAID without going offline, i.e switching off the volume
doing brick manipulations and switching it back on.
Cheers
Erekle
On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
>
> For any RAID 5 or 6 configuration I normally follow a simple gold
> rule which gave good results so far:
> - up to 4 disks RAID 5
> - 5 or more disks RAID 6
>
> However I didn't really understand well the recommendation to use any
> RAID with GlusterFS. I always thought that GlusteFS likes to work in
> JBOD mode and control the disks (bricks) directlly so you can create
> whatever distribution rule you wish, and if a single disk fails you
> just replace it and which obviously have the data replicated from
> another. The only downside of using in this way is that the
> replication data will be flow accross all servers but that is not
> much a big issue.
>
> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
>
> Thanks
> Regards
> Fernando
>
>
> On 07/08/2017 03:46, Devin Acosta wrote:
>>
>> Moacir,
>>
>> I have recently installed multiple Red Hat Virtualization hosts for
>> several different companies, and have dealt with the Red Hat Support
>> Team in depth about optimal configuration in regards to setting up
>> GlusterFS most efficiently and I wanted to share with you what I
>> learned.
>>
>> In general Red Hat Virtualization team frowns upon using each DISK
>> of the system as just a JBOD, sure there is some protection by
>> having the data replicated, however, the recommendation is to use
>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.
>>
>> Here is the direct quote from Red Hat when I asked about RAID and
>> Bricks:
>> /
>> /
>> /"A typical Gluster configuration would use RAID underneath the
>> bricks. RAID 6 is most typical as it gives you 2 disk failure
>> protection, but RAID 5 could be used too. Once you have the RAIDed
>> bricks, you'd then apply the desired replication on top of that. The
>> most popular way of doing this would be distributed replicated with
>> 2x replication. In general you'll get better performance with larger
>> bricks. 12 drives is often a sweet spot. Another option would be to
>> create a separate tier using all SSD’s.” /
>>
>> /In order to SSD tiering from my understanding you would need 1 x
>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be
>> distributed, replicated for the hot tier if not using NVME). So with
>> you only having 1 SSD drive in each server, I’d suggest maybe
>> looking into the NVME option. /
>> /
>> /
>> /Since your using only 3-servers, what I’d probably suggest is to do
>> (2 Replicas + Arbiter Node), this setup actually doesn’t require the
>> 3rd server to have big drives at all as it only stores meta-data
>> about the files and not actually a full copy. /
>> /
>> /
>> /Please see the attached document that was given to me by Red Hat to
>> get more information on this. Hope this information helps you./
>> /
>> /
>>
>> --
>>
>> Devin Acosta, RHCA, RHVCA
>> Red Hat Certified Architect
>>
>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
>> (moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>) wrote:
>>
>>> I am willing to assemble a oVirt "pod", made of 3 servers, each
>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers
>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to
>>> create a loop like a server triangle using the 40Gb NICs for
>>> virtualization files (VMs .qcow2) access and to move VMs around the
>>> pod (east /west traffic) while using the 10Gb interfaces for giving
>>> services to the outside world (north/south traffic).
>>>
>>>
>>> This said, my first question is: How should I deploy GlusterFS in
>>> such oVirt scenario? My questions are:
>>>
>>>
>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node,
>>> and then create a GlusterFS using them?
>>>
>>> 2 - Instead, should I create a JBOD array made of all server's disks?
>>>
>>> 3 - What is the best Gluster configuration to provide for HA while
>>> not consuming too much disk space?
>>>
>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and
>>> the virtualization environment, benefits from tiering when using a
>>> SSD disk? And yes, will Gluster do it by default or I have to
>>> configure it to do so?
>>>
>>>
>>> At the bottom line, what is the good practice for using GlusterFS
>>> in small pods for enterprises?
>>>
>>>
>>> You opinion/feedback will be really appreciated!
>>>
>>> Moacir
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--------------A5088546CDF99006314CCEE8
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Thanks for the clarification Erekle.</p>
<p>However I get surprised with this way of operating from GlusterFS
as it adds another layer of complexity to the system (either a
hardware or software RAID) before the gluster config and increase
the system's overall costs.<br>
</p>
<p>An important point to consider is: In RAID configuration you
already have space 'wasted' in order to build redundancy (either
RAID 1, 5, or 6). Then when you have GlusterFS on the top of
several RAIDs you have again more data replicated so you end up
with the same data consuming more space in a group of disks and
again on the top of several RAIDs depending on the Gluster
configuration you have (in a RAID 1 config the same data is
replicated 4 times).</p>
<p>Yet another downside of having a RAID (specially RAID 5 or 6) is
that it reduces considerably the write speeds as each group of
disks will end up having the write speed of a single disk as all
other disks of that group have to wait for each other to write as
well.<br>
</p>
<p>Therefore if Gluster already replicates data why does it create
this big pain you mentioned if the data is replicated somewhere
else, can still be retrieved to both serve clients and reconstruct
the equivalent disk when it is replaced ?</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 10:26, Erekle Magradze
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:aa829d07-fa77-3ed9-2500-e33cc01414b6@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Frenando,</p>
<p>Here is my experience, if you consider a particular hard drive
as a brick for gluster volume and it dies, i.e. it becomes not
accessible it's a huge hassle to discard that brick and exchange
with another one, since gluster some tries to access that broken
brick and it's causing (at least it cause for me) a big pain,
therefore it's better to have a RAID as brick, i.e. have RAID 1
(mirroring) for each brick, in this case if the disk is down you
can easily exchange it and rebuild the RAID without going
offline, i.e switching off the volume doing brick manipulations
and switching it back on.<br>
</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 03:04 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>For any RAID 5 or 6 configuration I normally follow a simple
gold rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the recommendation to
use any RAID with GlusterFS. I always thought that GlusteFS
likes to work in JBOD mode and control the disks (bricks)
directlly so you can create whatever distribution rule you
wish, and if a single disk fails you just replace it and which
obviously have the data replicated from another. The only
downside of using in this way is that the replication data
will be flow accross all servers but that is not much a big
issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and JBOD +
GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin Acosta
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">I have recently installed multiple Red
Hat Virtualization hosts for several different companies,
and have dealt with the Red Hat Support Team in depth
about optimal configuration in regards to setting up
GlusterFS most efficiently and I wanted to share with you
what I learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono">In general Red Hat Virtualization team
frowns upon using each DISK of the system as just a JBOD,
sure there is some protection by having the data
replicated, however, the recommendation is to use RAID 6
(preferred) or RAID-5, or at least RAID-1 at the very
least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input Mono"><br>
</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono">Here is the direct quote from Red Hat
when I asked about RAID and Bricks:</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>"A typical Gluster
configuration
would use RAID underneath the bricks. RAID 6 is most
typical as it gives you 2 disk failure protection, but
RAID 5 could be used too. Once you have the RAIDed
bricks, you'd then apply the desired replication on top
of that. The most popular way of doing this would be
distributed replicated with 2x replication. In general
you'll get better performance with larger bricks. 12
drives is often a sweet spot. Another option would be to
create a separate tier using all
SSD’s.” </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><br>
</div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>In order to SSD tiering from my
understanding you would need 1 x NVMe drive in each
server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME). So with
you only having 1 SSD drive in each server, I’d suggest
maybe looking into the NVME option. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Since your using only 3-servers,
what
I’d probably suggest is to do (2 Replicas + Arbiter
Node), this setup actually doesn’t require the 3rd
server to have big drives at all as it only stores
meta-data about the files and not actually a full
copy. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Please see the attached document
that
was given to me by Red Hat to get more information on
this. Hope this information helps you.</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature"
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir
Ferreira (<a href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite" class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt
"pod",
made of 3 servers, each with 2 CPU sockets of 12
cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is
to use GlusterFS to provide HA for the VMs. The
3 servers have a dual 40Gb NIC and a dual 10Gb
NIC. So my intention is to create a loop like a
server triangle using the 40Gb NICs for
virtualization files (VMs .qcow2) access and to
move VMs around the pod (east /west traffic)
while using the 10Gb interfaces for giving
services to the outside world (north/south
traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I
deploy GlusterFS in such oVirt scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on
each oVirt node, and then create a GlusterFS using
them?</p>
<p>2 - Instead, should I create a JBOD array made of
all server's disks?</p>
<p>3 - What is the best Gluster configuration to
provide for HA while not consuming too much disk
space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am
planning to build, and the virtualization
environment, benefits from tiering when using a
SSD disk? And yes, will Gluster do it by default
or I have to configure it to do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice for
using GlusterFS in small pods for enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>
--------------A5088546CDF99006314CCEE8--
9:43 p.m.
On Mon, Aug 7, 2017 at 6:41 PM, FERNANDO FREDIANI <fernando.frediani(a)upx.com
wrote:
Thanks for the clarification Erekle.
However I get surprised with this way of operating from GlusterFS as it
adds another layer of complexity to the system (either a hardware or
software RAID) before the gluster config and increase the system's overall
costs.
It does, but with HW based RAID it's not a big deal. The complexity is all
the stripe size math... which I personally don't like to calculate.
An important point to consider is: In RAID configuration you already
have
space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then
when you have GlusterFS on the top of several RAIDs you have again more
data replicated so you end up with the same data consuming more space in a
group of disks and again on the top of several RAIDs depending on the
Gluster configuration you have (in a RAID 1 config the same data is
replicated 4 times).
Yet another downside of having a RAID (specially RAID 5 or 6) is that it
reduces considerably the write speeds as each group of disks will end up
having the write speed of a single disk as all other disks of that group
have to wait for each other to write as well.
Therefore if Gluster already replicates data why does it create this big
pain you mentioned if the data is replicated somewhere else, can still be
retrieved to both serve clients and reconstruct the equivalent disk when it
is replaced ?
I think it's a matter of how fast you can replace a disk (over a long
weekend?), how reliably you can do it (please, don't pull the wrong disk!
I've seen it happening too many times!) and how much of a performance hit
are you willing to accept while in degraded mode (and how long it took to
detect it. HDDs, unlike SSDs, die slowly. At least when SSD dies, it dies a
quick and determined death. HDDs may accumulate errors and errors and still
function).
Y.
Fernando
On 07/08/2017 10:26, Erekle Magradze wrote:
Hi Frenando,
Here is my experience, if you consider a particular hard drive as a brick
for gluster volume and it dies, i.e. it becomes not accessible it's a huge
hassle to discard that brick and exchange with another one, since gluster
some tries to access that broken brick and it's causing (at least it cause
for me) a big pain, therefore it's better to have a RAID as brick, i.e.
have RAID 1 (mirroring) for each brick, in this case if the disk is down
you can easily exchange it and rebuild the RAID without going offline, i.e
switching off the volume doing brick manipulations and switching it back on.
Cheers
Erekle
On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
For any RAID 5 or 6 configuration I normally follow a simple gold rule
which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6
However I didn't really understand well the recommendation to use any RAID
with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode
and control the disks (bricks) directlly so you can create whatever
distribution rule you wish, and if a single disk fails you just replace it
and which obviously have the data replicated from another. The only
downside of using in this way is that the replication data will be flow
accross all servers but that is not much a big issue.
Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
Thanks
Regards
Fernando
On 07/08/2017 03:46, Devin Acosta wrote:
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for
several different companies, and have dealt with the Red Hat Support Team
in depth about optimal configuration in regards to setting up GlusterFS
most efficiently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the
system as just a JBOD, sure there is some protection by having the data
replicated, however, the recommendation is to use RAID 6 (preferred) or
RAID-5, or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
*"A typical Gluster configuration would use RAID underneath the bricks.
RAID 6 is most typical as it gives you 2 disk failure protection, but RAID
5 could be used too. Once you have the RAIDed bricks, you'd then apply the
desired replication on top of that. The most popular way of doing this
would be distributed replicated with 2x replication. In general you'll get
better performance with larger bricks. 12 drives is often a sweet spot.
Another option would be to create a separate tier using all SSD’s.” *
*In order to SSD tiering from my understanding you would need 1 x NVMe
drive in each server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME). So with you only having 1
SSD drive in each server, I’d suggest maybe looking into the NVME option. *
*Since your using only 3-servers, what I’d probably suggest is to do (2
Replicas + Arbiter Node), this setup actually doesn’t require the 3rd
server to have big drives at all as it only stores meta-data about the
files and not actually a full copy. *
*Please see the attached document that was given to me by Red Hat to get
more information on this. Hope this information helps you.*
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (
moacirferreira(a)hotmail.com) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
a dual 10Gb NIC. So my intention is to create a loop like a server triangle
using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
move VMs around the pod (east /west traffic) while using the 10Gb
interfaces for giving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
then create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:49 p.m.
--Apple-Mail=_912FAEC6-5476-4359-94B5-21FE877C0F94
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Le 7 ao=C3=BBt 2017 =C3=A0 17:41, FERNANDO FREDIANI =
<fernando.frediani(a)upx.com> a =C3=A9crit :
=20
Yet another downside of having a RAID (specially RAID 5 or 6) is that
=
it reduces considerably the write speeds as each group of disks will end =
up having the write speed of a single disk as all other disks of that =
group have to wait for each other to write as well.
=20
That's not true if you have medium to high range hardware raid. For =
example, HP Smart Array come with a flash cache of about 1 or 2 Gb that =
hides that from the OS.=
--Apple-Mail=_912FAEC6-5476-4359-94B5-21FE877C0F94
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">Le 7 ao=C3=BBt 2017 =C3=A0 17:41, FERNANDO FREDIANI <<a
=
href=3D"mailto:fernando.frediani@upx.com" =
class=3D"">fernando.frediani(a)upx.com</a>&gt; a =C3=A9crit
:</div><br =
class=3D"Apple-interchange-newline"><div class=3D""><p =
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" =
class=3D""></p></div></blockquote><br
class=3D""><blockquote type=3D"cite"=
class=3D""><div class=3D""><p style=3D"font-family:
Helvetica; =
font-size: 12px; font-style: normal; font-variant-caps: normal; =
font-weight: normal; letter-spacing: normal; orphans: auto; text-align: =
start; text-indent: 0px; text-transform: none; white-space: normal; =
widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
background-color: rgb(255, 255, 255);" class=3D"">Yet another downside
=
of having a RAID (specially RAID 5 or 6) is that it reduces considerably =
the write speeds as each group of disks will end up having the write =
speed of a single disk as all other disks of that group have to wait for =
each other to write as well.<br
class=3D""></p></div></blockquote><br =
class=3D""></div>That's not true if you have medium to high range
=
hardware raid. For example, HP Smart Array come with a flash cache of =
about 1 or 2 Gb that hides that from the OS.</body></html>=
--Apple-Mail=_912FAEC6-5476-4359-94B5-21FE877C0F94--
10:52 p.m.
This is a multi-part message in MIME format.
--------------2B4B839116333F84136665DD
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
What you mentioned is a specific case and not a generic situation. The
main point there is that RAID 5 or 6 impacts write performance compared
when you write to only 2 given disks at a time. That was the comparison
made.
Fernando
On 07/08/2017 16:49, Fabrice Bacchella wrote:
> Le 7 août 2017 à 17:41, FERNANDO FREDIANI <fernando.frediani(a)upx.com
> <mailto:fernando.frediani@upx.com>> a écrit :
>
> Yet another downside of having a RAID (specially RAID 5 or 6) is that
> it reduces considerably the write speeds as each group of disks will
> end up having the write speed of a single disk as all other disks of
> that group have to wait for each other to write as well.
>
That's not true if you have medium to high range hardware raid. For
example, HP Smart Array come with a flash cache of about 1 or 2 Gb
that hides that from the OS.
--------------2B4B839116333F84136665DD
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>What you mentioned is a specific case and not a generic
situation. The main point there is that RAID 5 or 6 impacts write
performance compared when you write to only 2 given disks at a
time. That was the comparison made.<br>
</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 16:49, Fabrice Bacchella
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:034E384B-4B32-4FBC-A760-404EB38EBC86@orange.fr">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">Le 7 août 2017 à 17:41, FERNANDO FREDIANI
<<a
href="mailto:fernando.frediani@upx.com" class=""
moz-do-not-send="true">fernando.frediani(a)upx.com</a>&gt; a
écrit :</div>
<br class="Apple-interchange-newline">
</blockquote>
<br class="">
<blockquote type="cite" class="">
<div class="">
<p style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans:
auto; text-align: start; text-indent: 0px; text-transform:
none; white-space: normal; widows: auto; word-spacing:
0px; -webkit-text-stroke-width: 0px; background-color:
rgb(255, 255, 255);" class="">Yet another downside of
having a RAID (specially RAID 5 or 6) is that it reduces
considerably the write speeds as each group of disks will
end up having the write speed of a single disk as all
other disks of that group have to wait for each other to
write as well.<br class="">
</p>
</div>
</blockquote>
<br class="">
</div>
That's not true if you have medium to high range hardware raid.
For example, HP Smart Array come with a flash cache of about 1 or
2 Gb that hides that from the OS.
</blockquote>
<br>
</body>
</html>
--------------2B4B839116333F84136665DD--
11:05 p.m.
This is a multi-part message in MIME format.
--------------A0081E07F314CF9B687DDA86
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi Franando,
So let's go with the following scenarios:
1. Let's say you have two servers (replication factor is 2), i.e. two
bricks per volume, in this case it is strongly recommended to have the
arbiter node, the metadata storage that will guarantee avoiding the
split brain situation, in this case for arbiter you don't even need a
disk with lots of space, it's enough to have a tiny ssd but hosted on a
separate server. Advantage of such setup is that you don't need the RAID
1 for each brick, you have the metadata information stored in arbiter
node and brick replacement is easy.
2. If you have odd number of bricks (let's say 3, i.e. replication
factor is 3) in your volume and you didn't create the arbiter node as
well as you didn't configure the quorum, in this case the entire load
for keeping the consistency of the volume resides on all 3 servers, each
of them is important and each brick contains key information, they need
to cross-check each other (that's what people usually do with the first
try of gluster :) ), in this case replacing a brick is a big pain and in
this case RAID 1 is a good option to have (that's the disadvantage, i.e.
loosing the space and not having the JBOD option) advantage is that you
don't have the to have additional arbiter node.
3. You have odd number of bricks and configured arbiter node, in this
case you can easily go with JBOD, however a good practice would be to
have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly
sufficient for volumes with 10s of TB-s in size.)
That's basically it
The rest about the reliability and setup scenarios you can find in
gluster documentation, especially look for quorum and arbiter node
configs+options.
Cheers
Erekle
P.S. What I was mentioning, regarding a good practice is mostly related
to the operations of gluster not installation or deployment, i.e. not
the conceptual understanding of gluster (conceptually it's a JBOD system).
On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:
Thanks for the clarification Erekle.
However I get surprised with this way of operating from GlusterFS as
it adds another layer of complexity to the system (either a hardware
or software RAID) before the gluster config and increase the system's
overall costs.
An important point to consider is: In RAID configuration you already
have space 'wasted' in order to build redundancy (either RAID 1, 5, or
6). Then when you have GlusterFS on the top of several RAIDs you have
again more data replicated so you end up with the same data consuming
more space in a group of disks and again on the top of several RAIDs
depending on the Gluster configuration you have (in a RAID 1 config
the same data is replicated 4 times).
Yet another downside of having a RAID (specially RAID 5 or 6) is that
it reduces considerably the write speeds as each group of disks will
end up having the write speed of a single disk as all other disks of
that group have to wait for each other to write as well.
Therefore if Gluster already replicates data why does it create this
big pain you mentioned if the data is replicated somewhere else, can
still be retrieved to both serve clients and reconstruct the
equivalent disk when it is replaced ?
Fernando
On 07/08/2017 10:26, Erekle Magradze wrote:
>
> Hi Frenando,
>
> Here is my experience, if you consider a particular hard drive as a
> brick for gluster volume and it dies, i.e. it becomes not accessible
> it's a huge hassle to discard that brick and exchange with another
> one, since gluster some tries to access that broken brick and it's
> causing (at least it cause for me) a big pain, therefore it's better
> to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick,
> in this case if the disk is down you can easily exchange it and
> rebuild the RAID without going offline, i.e switching off the volume
> doing brick manipulations and switching it back on.
>
> Cheers
>
> Erekle
>
>
> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
>>
>> For any RAID 5 or 6 configuration I normally follow a simple gold
>> rule which gave good results so far:
>> - up to 4 disks RAID 5
>> - 5 or more disks RAID 6
>>
>> However I didn't really understand well the recommendation to use
>> any RAID with GlusterFS. I always thought that GlusteFS likes to
>> work in JBOD mode and control the disks (bricks) directlly so you
>> can create whatever distribution rule you wish, and if a single disk
>> fails you just replace it and which obviously have the data
>> replicated from another. The only downside of using in this way is
>> that the replication data will be flow accross all servers but that
>> is not much a big issue.
>>
>> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
>>
>> Thanks
>> Regards
>> Fernando
>>
>>
>> On 07/08/2017 03:46, Devin Acosta wrote:
>>>
>>> Moacir,
>>>
>>> I have recently installed multiple Red Hat Virtualization hosts for
>>> several different companies, and have dealt with the Red Hat
>>> Support Team in depth about optimal configuration in regards to
>>> setting up GlusterFS most efficiently and I wanted to share with
>>> you what I learned.
>>>
>>> In general Red Hat Virtualization team frowns upon using each DISK
>>> of the system as just a JBOD, sure there is some protection by
>>> having the data replicated, however, the recommendation is to use
>>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.
>>>
>>> Here is the direct quote from Red Hat when I asked about RAID and
>>> Bricks:
>>> /
>>> /
>>> /"A typical Gluster configuration would use RAID underneath the
>>> bricks. RAID 6 is most typical as it gives you 2 disk failure
>>> protection, but RAID 5 could be used too. Once you have the RAIDed
>>> bricks, you'd then apply the desired replication on top of that.
>>> The most popular way of doing this would be distributed replicated
>>> with 2x replication. In general you'll get better performance with
>>> larger bricks. 12 drives is often a sweet spot. Another option
>>> would be to create a separate tier using all SSD’s.” /
>>>
>>> /In order to SSD tiering from my understanding you would need 1 x
>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be
>>> distributed, replicated for the hot tier if not using NVME). So
>>> with you only having 1 SSD drive in each server, I’d suggest maybe
>>> looking into the NVME option. /
>>> /
>>> /
>>> /Since your using only 3-servers, what I’d probably suggest is to
>>> do (2 Replicas + Arbiter Node), this setup actually doesn’t require
>>> the 3rd server to have big drives at all as it only stores
>>> meta-data about the files and not actually a full copy. /
>>> /
>>> /
>>> /Please see the attached document that was given to me by Red Hat
>>> to get more information on this. Hope this information helps you./
>>> /
>>> /
>>>
>>> --
>>>
>>> Devin Acosta, RHCA, RHVCA
>>> Red Hat Certified Architect
>>>
>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
>>> (moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>)
wrote:
>>>
>>>> I am willing to assemble a oVirt "pod", made of 3 servers, each
>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
>>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers
>>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to
>>>> create a loop like a server triangle using the 40Gb NICs for
>>>> virtualization files (VMs .qcow2) access and to move VMs around
>>>> the pod (east /west traffic) while using the 10Gb interfaces for
>>>> giving services to the outside world (north/south traffic).
>>>>
>>>>
>>>> This said, my first question is: How should I deploy GlusterFS in
>>>> such oVirt scenario? My questions are:
>>>>
>>>>
>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node,
>>>> and then create a GlusterFS using them?
>>>>
>>>> 2 - Instead, should I create a JBOD array made of all server's
disks?
>>>>
>>>> 3 - What is the best Gluster configuration to provide for HA while
>>>> not consuming too much disk space?
>>>>
>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and
>>>> the virtualization environment, benefits from tiering when using a
>>>> SSD disk? And yes, will Gluster do it by default or I have to
>>>> configure it to do so?
>>>>
>>>>
>>>> At the bottom line, what is the good practice for using GlusterFS
>>>> in small pods for enterprises?
>>>>
>>>>
>>>> You opinion/feedback will be really appreciated!
>>>>
>>>> Moacir
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail erekle.magradze(a)recogizer.de
Web: www.recogizer.com
Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/
Folgen Sie uns auf Twitter https://twitter.com/recogizer
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.
--------------A0081E07F314CF9B687DDA86
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Franando,</p>
<p>So let's go with the following scenarios:</p>
<p>1. Let's say you have two servers (replication factor is 2), i.e.
two bricks per volume, in this case it is strongly recommended to
have the arbiter node, the metadata storage that will guarantee
avoiding the split brain situation, in this case for arbiter you
don't even need a disk with lots of space, it's enough to have a
tiny ssd but hosted on a separate server. Advantage of such setup
is that you don't need the RAID 1 for each brick, you have the
metadata information stored in arbiter node and brick replacement
is easy.</p>
<p>2. If you have odd number of bricks (let's say 3, i.e.
replication factor is 3) in your volume and you didn't create the
arbiter node as well as you didn't configure the quorum, in this
case the entire load for keeping the consistency of the volume
resides on all 3 servers, each of them is important and each brick
contains key information, they need to cross-check each other
(that's what people usually do with the first try of gluster :) ),
in this case replacing a brick is a big pain and in this case RAID
1 is a good option to have (that's the disadvantage, i.e. loosing
the space and not having the JBOD option) advantage is that you
don't have the to have additional arbiter node.</p>
<p>3. You have odd number of bricks and configured arbiter node, in
this case you can easily go with JBOD, however a good practice
would be to have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar
perfectly sufficient for volumes with 10s of TB-s in size.)</p>
<p>That's basically it</p>
<p>The rest about the reliability and setup scenarios you can find
in gluster documentation, especially look for quorum and arbiter
node configs+options.</p>
<p>Cheers</p>
<p>Erekle</p>
<p>P.S. What I was mentioning, regarding a good practice is mostly
related to the operations of gluster not installation or
deployment, i.e. not the conceptual understanding of gluster
(conceptually it's a JBOD system).<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 05:41 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Thanks for the clarification Erekle.</p>
<p>However I get surprised with this way of operating from
GlusterFS as it adds another layer of complexity to the system
(either a hardware or software RAID) before the gluster config
and increase the system's overall costs.<br>
</p>
<p>An important point to consider is: In RAID configuration you
already have space 'wasted' in order to build redundancy (either
RAID 1, 5, or 6). Then when you have GlusterFS on the top of
several RAIDs you have again more data replicated so you end up
with the same data consuming more space in a group of disks and
again on the top of several RAIDs depending on the Gluster
configuration you have (in a RAID 1 config the same data is
replicated 4 times).</p>
<p>Yet another downside of having a RAID (specially RAID 5 or 6)
is that it reduces considerably the write speeds as each group
of disks will end up having the write speed of a single disk as
all other disks of that group have to wait for each other to
write as well.<br>
</p>
<p>Therefore if Gluster already replicates data why does it create
this big pain you mentioned if the data is replicated somewhere
else, can still be retrieved to both serve clients and
reconstruct the equivalent disk when it is replaced ?</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 10:26, Erekle Magradze
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:aa829d07-fa77-3ed9-2500-e33cc01414b6@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Frenando,</p>
<p>Here is my experience, if you consider a particular hard
drive as a brick for gluster volume and it dies, i.e. it
becomes not accessible it's a huge hassle to discard that
brick and exchange with another one, since gluster some tries
to access that broken brick and it's causing (at least it
cause for me) a big pain, therefore it's better to have a RAID
as brick, i.e. have RAID 1 (mirroring) for each brick, in this
case if the disk is down you can easily exchange it and
rebuild the RAID without going offline, i.e switching off the
volume doing brick manipulations and switching it back on.<br>
</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 03:04 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>For any RAID 5 or 6 configuration I normally follow a
simple gold rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the recommendation
to use any RAID with GlusterFS. I always thought that
GlusteFS likes to work in JBOD mode and control the disks
(bricks) directlly so you can create whatever distribution
rule you wish, and if a single disk fails you just replace
it and which obviously have the data replicated from
another. The only downside of using in this way is that the
replication data will be flow accross all servers but that
is not much a big issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and JBOD
+ GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin Acosta
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono">I have recently installed multiple Red Hat
Virtualization hosts for several different companies,
and have dealt with the Red Hat Support Team in depth
about optimal configuration in regards to setting up
GlusterFS most efficiently and I wanted to share with
you what I learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono">In general Red Hat Virtualization team frowns upon
using each DISK of the system as just a JBOD, sure there
is some protection by having the data replicated,
however, the recommendation is to use RAID 6 (preferred)
or RAID-5, or at least RAID-1 at the very least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono">Here is the direct quote from Red Hat
when I asked about RAID and Bricks:</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>"A typical Gluster
configuration
would use RAID underneath the bricks. RAID 6 is most
typical as it gives you 2 disk failure protection, but
RAID 5 could be used too. Once you have the RAIDed
bricks, you'd then apply the desired replication on
top of that. The most popular way of doing this would
be distributed replicated with 2x replication. In
general you'll get better performance with larger
bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all
SSD’s.” </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><br>
</div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>In order to SSD tiering from my
understanding you would need 1 x NVMe drive in each
server, or 4 x SSD hot tier (it needs to be
distributed, replicated for the hot tier if not using
NVME). So with you only having 1 SSD drive in each
server, I’d suggest maybe looking into the NVME
option. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Since your using only 3-servers,
what I’d probably suggest is to do (2 Replicas +
Arbiter Node), this setup actually doesn’t require the
3rd server to have big drives at all as it only stores
meta-data about the files and not actually a full
copy. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Please see the attached document
that was given to me by Red Hat to get more
information on this. Hope this information helps
you.</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature"
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM,
Moacir Ferreira (<a
href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite"
class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt
"pod",
made of 3 servers, each with 2 CPU sockets of
12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
idea is to use GlusterFS to provide HA for the
VMs. The 3 servers have a dual 40Gb NIC and a
dual 10Gb NIC. So my intention is to create a
loop like a server triangle using the 40Gb
NICs for virtualization files (VMs .qcow2)
access and to move VMs around the pod (east
/west traffic) while using the 10Gb interfaces
for giving services to the outside world
(north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I
deploy GlusterFS in such oVirt scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one
on each oVirt node, and then create a GlusterFS
using them?</p>
<p>2 - Instead, should I create a JBOD array made
of all server's disks?</p>
<p>3 - What is the best Gluster configuration to
provide for HA while not consuming too much disk
space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am
planning to build, and the virtualization
environment, benefits from tiering when using a
SSD disk? And yes, will Gluster do it by default
or I have to configure it to do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice
for using GlusterFS in small pods for
enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really
appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail <a class="moz-txt-link-abbreviated"
href="mailto:erekle.magradze@recogizer.de">erekle.magradze@recogizer.de</a>
Web: <a class="moz-txt-link-abbreviated"
href="http://www.recogizer.com">www.recogizer.com</a>
Recogizer auf LinkedIn <a class="moz-txt-link-freetext"
href="https://www.linkedin.com/company-beta/10039182/">https...
Folgen Sie uns auf Twitter <a class="moz-txt-link-freetext"
href="https://twitter.com/recogizer">https://twitter.com/rec...
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.</pre>
</body>
</html>
--------------A0081E07F314CF9B687DDA86--
11:07 p.m.
This is a multi-part message in MIME format.
--------------0DC09401E4EDC25E68CD6208
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi Fernando (sorry for misspelling your name, I used a different keyboard),
So let's go with the following scenarios:
1. Let's say you have two servers (replication factor is 2), i.e. two
bricks per volume, in this case it is strongly recommended to have the
arbiter node, the metadata storage that will guarantee avoiding the
split brain situation, in this case for arbiter you don't even need a
disk with lots of space, it's enough to have a tiny ssd but hosted on a
separate server. Advantage of such setup is that you don't need the RAID
1 for each brick, you have the metadata information stored in arbiter
node and brick replacement is easy.
2. If you have odd number of bricks (let's say 3, i.e. replication
factor is 3) in your volume and you didn't create the arbiter node as
well as you didn't configure the quorum, in this case the entire load
for keeping the consistency of the volume resides on all 3 servers, each
of them is important and each brick contains key information, they need
to cross-check each other (that's what people usually do with the first
try of gluster :) ), in this case replacing a brick is a big pain and in
this case RAID 1 is a good option to have (that's the disadvantage, i.e.
loosing the space and not having the JBOD option) advantage is that you
don't have the to have additional arbiter node.
3. You have odd number of bricks and configured arbiter node, in this
case you can easily go with JBOD, however a good practice would be to
have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly
sufficient for volumes with 10s of TB-s in size.)
That's basically it
The rest about the reliability and setup scenarios you can find in
gluster documentation, especially look for quorum and arbiter node
configs+options.
Cheers
Erekle
P.S. What I was mentioning, regarding a good practice is mostly related
to the operations of gluster not installation or deployment, i.e. not
the conceptual understanding of gluster (conceptually it's a JBOD system).
On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:
Thanks for the clarification Erekle.
However I get surprised with this way of operating from GlusterFS as
it adds another layer of complexity to the system (either a hardware
or software RAID) before the gluster config and increase the system's
overall costs.
An important point to consider is: In RAID configuration you already
have space 'wasted' in order to build redundancy (either RAID 1, 5, or
6). Then when you have GlusterFS on the top of several RAIDs you have
again more data replicated so you end up with the same data consuming
more space in a group of disks and again on the top of several RAIDs
depending on the Gluster configuration you have (in a RAID 1 config
the same data is replicated 4 times).
Yet another downside of having a RAID (specially RAID 5 or 6) is that
it reduces considerably the write speeds as each group of disks will
end up having the write speed of a single disk as all other disks of
that group have to wait for each other to write as well.
Therefore if Gluster already replicates data why does it create this
big pain you mentioned if the data is replicated somewhere else, can
still be retrieved to both serve clients and reconstruct the
equivalent disk when it is replaced ?
Fernando
On 07/08/2017 10:26, Erekle Magradze wrote:
>
> Hi Frenando,
>
> Here is my experience, if you consider a particular hard drive as a
> brick for gluster volume and it dies, i.e. it becomes not accessible
> it's a huge hassle to discard that brick and exchange with another
> one, since gluster some tries to access that broken brick and it's
> causing (at least it cause for me) a big pain, therefore it's better
> to have a RAID as brick, i.e. have RAID 1 (mirroring) for each brick,
> in this case if the disk is down you can easily exchange it and
> rebuild the RAID without going offline, i.e switching off the volume
> doing brick manipulations and switching it back on.
>
> Cheers
>
> Erekle
>
>
> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
>>
>> For any RAID 5 or 6 configuration I normally follow a simple gold
>> rule which gave good results so far:
>> - up to 4 disks RAID 5
>> - 5 or more disks RAID 6
>>
>> However I didn't really understand well the recommendation to use
>> any RAID with GlusterFS. I always thought that GlusteFS likes to
>> work in JBOD mode and control the disks (bricks) directlly so you
>> can create whatever distribution rule you wish, and if a single disk
>> fails you just replace it and which obviously have the data
>> replicated from another. The only downside of using in this way is
>> that the replication data will be flow accross all servers but that
>> is not much a big issue.
>>
>> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
>>
>> Thanks
>> Regards
>> Fernando
>>
>>
>> On 07/08/2017 03:46, Devin Acosta wrote:
>>>
>>> Moacir,
>>>
>>> I have recently installed multiple Red Hat Virtualization hosts for
>>> several different companies, and have dealt with the Red Hat
>>> Support Team in depth about optimal configuration in regards to
>>> setting up GlusterFS most efficiently and I wanted to share with
>>> you what I learned.
>>>
>>> In general Red Hat Virtualization team frowns upon using each DISK
>>> of the system as just a JBOD, sure there is some protection by
>>> having the data replicated, however, the recommendation is to use
>>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.
>>>
>>> Here is the direct quote from Red Hat when I asked about RAID and
>>> Bricks:
>>> /
>>> /
>>> /"A typical Gluster configuration would use RAID underneath the
>>> bricks. RAID 6 is most typical as it gives you 2 disk failure
>>> protection, but RAID 5 could be used too. Once you have the RAIDed
>>> bricks, you'd then apply the desired replication on top of that.
>>> The most popular way of doing this would be distributed replicated
>>> with 2x replication. In general you'll get better performance with
>>> larger bricks. 12 drives is often a sweet spot. Another option
>>> would be to create a separate tier using all SSD’s.” /
>>>
>>> /In order to SSD tiering from my understanding you would need 1 x
>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be
>>> distributed, replicated for the hot tier if not using NVME). So
>>> with you only having 1 SSD drive in each server, I’d suggest maybe
>>> looking into the NVME option. /
>>> /
>>> /
>>> /Since your using only 3-servers, what I’d probably suggest is to
>>> do (2 Replicas + Arbiter Node), this setup actually doesn’t require
>>> the 3rd server to have big drives at all as it only stores
>>> meta-data about the files and not actually a full copy. /
>>> /
>>> /
>>> /Please see the attached document that was given to me by Red Hat
>>> to get more information on this. Hope this information helps you./
>>> /
>>> /
>>>
>>> --
>>>
>>> Devin Acosta, RHCA, RHVCA
>>> Red Hat Certified Architect
>>>
>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
>>> (moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>)
wrote:
>>>
>>>> I am willing to assemble a oVirt "pod", made of 3 servers, each
>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
>>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers
>>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to
>>>> create a loop like a server triangle using the 40Gb NICs for
>>>> virtualization files (VMs .qcow2) access and to move VMs around
>>>> the pod (east /west traffic) while using the 10Gb interfaces for
>>>> giving services to the outside world (north/south traffic).
>>>>
>>>>
>>>> This said, my first question is: How should I deploy GlusterFS in
>>>> such oVirt scenario? My questions are:
>>>>
>>>>
>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node,
>>>> and then create a GlusterFS using them?
>>>>
>>>> 2 - Instead, should I create a JBOD array made of all server's
disks?
>>>>
>>>> 3 - What is the best Gluster configuration to provide for HA while
>>>> not consuming too much disk space?
>>>>
>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and
>>>> the virtualization environment, benefits from tiering when using a
>>>> SSD disk? And yes, will Gluster do it by default or I have to
>>>> configure it to do so?
>>>>
>>>>
>>>> At the bottom line, what is the good practice for using GlusterFS
>>>> in small pods for enterprises?
>>>>
>>>>
>>>> You opinion/feedback will be really appreciated!
>>>>
>>>> Moacir
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail erekle.magradze(a)recogizer.de
Web: www.recogizer.com
Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/
Folgen Sie uns auf Twitter https://twitter.com/recogizer
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.
--------------0DC09401E4EDC25E68CD6208
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Fernando (sorry for misspelling your name, I used a different
keyboard),</p>
<p>So let's go with the following scenarios:</p>
<p>1. Let's say you have two servers (replication factor is 2), i.e.
two bricks per volume, in this case it is strongly recommended to
have the arbiter node, the metadata storage that will guarantee
avoiding the split brain situation, in this case for arbiter you
don't even need a disk with lots of space, it's enough to have a
tiny ssd but hosted on a separate server. Advantage of such setup
is that you don't need the RAID 1 for each brick, you have the
metadata information stored in arbiter node and brick replacement
is easy.</p>
<p>2. If you have odd number of bricks (let's say 3, i.e.
replication factor is 3) in your volume and you didn't create the
arbiter node as well as you didn't configure the quorum, in this
case the entire load for keeping the consistency of the volume
resides on all 3 servers, each of them is important and each brick
contains key information, they need to cross-check each other
(that's what people usually do with the first try of gluster :) ),
in this case replacing a brick is a big pain and in this case RAID
1 is a good option to have (that's the disadvantage, i.e. loosing
the space and not having the JBOD option) advantage is that you
don't have the to have additional arbiter node.</p>
<p>3. You have odd number of bricks and configured arbiter node, in
this case you can easily go with JBOD, however a good practice
would be to have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar
perfectly sufficient for volumes with 10s of TB-s in size.)</p>
<p>That's basically it</p>
<p>The rest about the reliability and setup scenarios you can find
in gluster documentation, especially look for quorum and arbiter
node configs+options.</p>
<p>Cheers</p>
<p>Erekle</p>
P.S. What I was mentioning, regarding a good practice is mostly
related to the operations of gluster not installation or deployment,
i.e. not the conceptual understanding of gluster (conceptually it's
a JBOD system).<br>
<br>
<div class="moz-cite-prefix">On 08/07/2017 05:41 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Thanks for the clarification Erekle.</p>
<p>However I get surprised with this way of operating from
GlusterFS as it adds another layer of complexity to the system
(either a hardware or software RAID) before the gluster config
and increase the system's overall costs.<br>
</p>
<p>An important point to consider is: In RAID configuration you
already have space 'wasted' in order to build redundancy (either
RAID 1, 5, or 6). Then when you have GlusterFS on the top of
several RAIDs you have again more data replicated so you end up
with the same data consuming more space in a group of disks and
again on the top of several RAIDs depending on the Gluster
configuration you have (in a RAID 1 config the same data is
replicated 4 times).</p>
<p>Yet another downside of having a RAID (specially RAID 5 or 6)
is that it reduces considerably the write speeds as each group
of disks will end up having the write speed of a single disk as
all other disks of that group have to wait for each other to
write as well.<br>
</p>
<p>Therefore if Gluster already replicates data why does it create
this big pain you mentioned if the data is replicated somewhere
else, can still be retrieved to both serve clients and
reconstruct the equivalent disk when it is replaced ?</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 10:26, Erekle Magradze
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:aa829d07-fa77-3ed9-2500-e33cc01414b6@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Frenando,</p>
<p>Here is my experience, if you consider a particular hard
drive as a brick for gluster volume and it dies, i.e. it
becomes not accessible it's a huge hassle to discard that
brick and exchange with another one, since gluster some tries
to access that broken brick and it's causing (at least it
cause for me) a big pain, therefore it's better to have a RAID
as brick, i.e. have RAID 1 (mirroring) for each brick, in this
case if the disk is down you can easily exchange it and
rebuild the RAID without going offline, i.e switching off the
volume doing brick manipulations and switching it back on.<br>
</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 03:04 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>For any RAID 5 or 6 configuration I normally follow a
simple gold rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the recommendation
to use any RAID with GlusterFS. I always thought that
GlusteFS likes to work in JBOD mode and control the disks
(bricks) directlly so you can create whatever distribution
rule you wish, and if a single disk fails you just replace
it and which obviously have the data replicated from
another. The only downside of using in this way is that the
replication data will be flow accross all servers but that
is not much a big issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and JBOD
+ GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin Acosta
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono">I have recently installed multiple Red Hat
Virtualization hosts for several different companies,
and have dealt with the Red Hat Support Team in depth
about optimal configuration in regards to setting up
GlusterFS most efficiently and I wanted to share with
you what I learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono">In general Red Hat Virtualization team frowns upon
using each DISK of the system as just a JBOD, sure there
is some protection by having the data replicated,
however, the recommendation is to use RAID 6 (preferred)
or RAID-5, or at least RAID-1 at the very least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono">Here is the direct quote from Red Hat
when I asked about RAID and Bricks:</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>"A typical Gluster
configuration
would use RAID underneath the bricks. RAID 6 is most
typical as it gives you 2 disk failure protection, but
RAID 5 could be used too. Once you have the RAIDed
bricks, you'd then apply the desired replication on
top of that. The most popular way of doing this would
be distributed replicated with 2x replication. In
general you'll get better performance with larger
bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all
SSD’s.” </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><br>
</div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>In order to SSD tiering from my
understanding you would need 1 x NVMe drive in each
server, or 4 x SSD hot tier (it needs to be
distributed, replicated for the hot tier if not using
NVME). So with you only having 1 SSD drive in each
server, I’d suggest maybe looking into the NVME
option. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Since your using only 3-servers,
what I’d probably suggest is to do (2 Replicas +
Arbiter Node), this setup actually doesn’t require the
3rd server to have big drives at all as it only stores
meta-data about the files and not actually a full
copy. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Please see the attached document
that was given to me by Red Hat to get more
information on this. Hope this information helps
you.</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature"
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM,
Moacir Ferreira (<a
href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite"
class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt
"pod",
made of 3 servers, each with 2 CPU sockets of
12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
idea is to use GlusterFS to provide HA for the
VMs. The 3 servers have a dual 40Gb NIC and a
dual 10Gb NIC. So my intention is to create a
loop like a server triangle using the 40Gb
NICs for virtualization files (VMs .qcow2)
access and to move VMs around the pod (east
/west traffic) while using the 10Gb interfaces
for giving services to the outside world
(north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I
deploy GlusterFS in such oVirt scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one
on each oVirt node, and then create a GlusterFS
using them?</p>
<p>2 - Instead, should I create a JBOD array made
of all server's disks?</p>
<p>3 - What is the best Gluster configuration to
provide for HA while not consuming too much disk
space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am
planning to build, and the virtualization
environment, benefits from tiering when using a
SSD disk? And yes, will Gluster do it by default
or I have to configure it to do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice
for using GlusterFS in small pods for
enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really
appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail <a class="moz-txt-link-abbreviated"
href="mailto:erekle.magradze@recogizer.de">erekle.magradze@recogizer.de</a>
Web: <a class="moz-txt-link-abbreviated"
href="http://www.recogizer.com">www.recogizer.com</a>
Recogizer auf LinkedIn <a class="moz-txt-link-freetext"
href="https://www.linkedin.com/company-beta/10039182/">https...
Folgen Sie uns auf Twitter <a class="moz-txt-link-freetext"
href="https://twitter.com/recogizer">https://twitter.com/rec...
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.</pre>
</body>
</html>
--------------0DC09401E4EDC25E68CD6208--
12:03 a.m.
This is a multi-part message in MIME format.
--------------D415F827B0DC677345E35156
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Thanks for the detailed answer Erekle.
I conclude that it is worth in any scenario to have a arbiter node in
order to avoid wasting more disk space to RAID X + Gluster Replication
on the top of it. The cost seems much lower if you consider running
costs of the whole storage and compare it with the cost to build the
arbiter node. Even having a fully redundant arbiter service with 2 nodes
would make it wort on a larger deployment.
Regards
Fernando
On 07/08/2017 17:07, Erekle Magradze wrote:
Hi Fernando (sorry for misspelling your name, I used a different
keyboard),
So let's go with the following scenarios:
1. Let's say you have two servers (replication factor is 2), i.e. two
bricks per volume, in this case it is strongly recommended to have the
arbiter node, the metadata storage that will guarantee avoiding the
split brain situation, in this case for arbiter you don't even need a
disk with lots of space, it's enough to have a tiny ssd but hosted on
a separate server. Advantage of such setup is that you don't need the
RAID 1 for each brick, you have the metadata information stored in
arbiter node and brick replacement is easy.
2. If you have odd number of bricks (let's say 3, i.e. replication
factor is 3) in your volume and you didn't create the arbiter node as
well as you didn't configure the quorum, in this case the entire load
for keeping the consistency of the volume resides on all 3 servers,
each of them is important and each brick contains key information,
they need to cross-check each other (that's what people usually do
with the first try of gluster :) ), in this case replacing a brick is
a big pain and in this case RAID 1 is a good option to have (that's
the disadvantage, i.e. loosing the space and not having the JBOD
option) advantage is that you don't have the to have additional
arbiter node.
3. You have odd number of bricks and configured arbiter node, in this
case you can easily go with JBOD, however a good practice would be to
have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly
sufficient for volumes with 10s of TB-s in size.)
That's basically it
The rest about the reliability and setup scenarios you can find in
gluster documentation, especially look for quorum and arbiter node
configs+options.
Cheers
Erekle
P.S. What I was mentioning, regarding a good practice is mostly
related to the operations of gluster not installation or deployment,
i.e. not the conceptual understanding of gluster (conceptually it's a
JBOD system).
On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:
>
> Thanks for the clarification Erekle.
>
> However I get surprised with this way of operating from GlusterFS as
> it adds another layer of complexity to the system (either a hardware
> or software RAID) before the gluster config and increase the system's
> overall costs.
>
> An important point to consider is: In RAID configuration you already
> have space 'wasted' in order to build redundancy (either RAID 1, 5,
> or 6). Then when you have GlusterFS on the top of several RAIDs you
> have again more data replicated so you end up with the same data
> consuming more space in a group of disks and again on the top of
> several RAIDs depending on the Gluster configuration you have (in a
> RAID 1 config the same data is replicated 4 times).
>
> Yet another downside of having a RAID (specially RAID 5 or 6) is that
> it reduces considerably the write speeds as each group of disks will
> end up having the write speed of a single disk as all other disks of
> that group have to wait for each other to write as well.
>
> Therefore if Gluster already replicates data why does it create this
> big pain you mentioned if the data is replicated somewhere else, can
> still be retrieved to both serve clients and reconstruct the
> equivalent disk when it is replaced ?
>
> Fernando
>
>
> On 07/08/2017 10:26, Erekle Magradze wrote:
>>
>> Hi Frenando,
>>
>> Here is my experience, if you consider a particular hard drive as a
>> brick for gluster volume and it dies, i.e. it becomes not accessible
>> it's a huge hassle to discard that brick and exchange with another
>> one, since gluster some tries to access that broken brick and it's
>> causing (at least it cause for me) a big pain, therefore it's better
>> to have a RAID as brick, i.e. have RAID 1 (mirroring) for each
>> brick, in this case if the disk is down you can easily exchange it
>> and rebuild the RAID without going offline, i.e switching off the
>> volume doing brick manipulations and switching it back on.
>>
>> Cheers
>>
>> Erekle
>>
>>
>> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
>>>
>>> For any RAID 5 or 6 configuration I normally follow a simple gold
>>> rule which gave good results so far:
>>> - up to 4 disks RAID 5
>>> - 5 or more disks RAID 6
>>>
>>> However I didn't really understand well the recommendation to use
>>> any RAID with GlusterFS. I always thought that GlusteFS likes to
>>> work in JBOD mode and control the disks (bricks) directlly so you
>>> can create whatever distribution rule you wish, and if a single
>>> disk fails you just replace it and which obviously have the data
>>> replicated from another. The only downside of using in this way is
>>> that the replication data will be flow accross all servers but that
>>> is not much a big issue.
>>>
>>> Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
>>>
>>> Thanks
>>> Regards
>>> Fernando
>>>
>>>
>>> On 07/08/2017 03:46, Devin Acosta wrote:
>>>>
>>>> Moacir,
>>>>
>>>> I have recently installed multiple Red Hat Virtualization hosts
>>>> for several different companies, and have dealt with the Red Hat
>>>> Support Team in depth about optimal configuration in regards to
>>>> setting up GlusterFS most efficiently and I wanted to share with
>>>> you what I learned.
>>>>
>>>> In general Red Hat Virtualization team frowns upon using each DISK
>>>> of the system as just a JBOD, sure there is some protection by
>>>> having the data replicated, however, the recommendation is to use
>>>> RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very least.
>>>>
>>>> Here is the direct quote from Red Hat when I asked about RAID and
>>>> Bricks:
>>>> /
>>>> /
>>>> /"A typical Gluster configuration would use RAID underneath the
>>>> bricks. RAID 6 is most typical as it gives you 2 disk failure
>>>> protection, but RAID 5 could be used too. Once you have the RAIDed
>>>> bricks, you'd then apply the desired replication on top of that.
>>>> The most popular way of doing this would be distributed replicated
>>>> with 2x replication. In general you'll get better performance with
>>>> larger bricks. 12 drives is often a sweet spot. Another option
>>>> would be to create a separate tier using all SSD’s.” /
>>>>
>>>> /In order to SSD tiering from my understanding you would need 1 x
>>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be
>>>> distributed, replicated for the hot tier if not using NVME). So
>>>> with you only having 1 SSD drive in each server, I’d suggest maybe
>>>> looking into the NVME option. /
>>>> /
>>>> /
>>>> /Since your using only 3-servers, what I’d probably suggest is to
>>>> do (2 Replicas + Arbiter Node), this setup actually doesn’t
>>>> require the 3rd server to have big drives at all as it only stores
>>>> meta-data about the files and not actually a full copy. /
>>>> /
>>>> /
>>>> /Please see the attached document that was given to me by Red Hat
>>>> to get more information on this. Hope this information helps you./
>>>> /
>>>> /
>>>>
>>>> --
>>>>
>>>> Devin Acosta, RHCA, RHVCA
>>>> Red Hat Certified Architect
>>>>
>>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
>>>> (moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>)
>>>> wrote:
>>>>
>>>>> I am willing to assemble a oVirt "pod", made of 3 servers,
each
>>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
>>>>> idea is to use GlusterFS to provide HA for the VMs. The 3 servers
>>>>> have a dual 40Gb NIC and a dual 10Gb NIC. So my intention is to
>>>>> create a loop like a server triangle using the 40Gb NICs for
>>>>> virtualization files (VMs .qcow2) access and to move VMs around
>>>>> the pod (east /west traffic) while using the 10Gb interfaces for
>>>>> giving services to the outside world (north/south traffic).
>>>>>
>>>>>
>>>>> This said, my first question is: How should I deploy GlusterFS in
>>>>> such oVirt scenario? My questions are:
>>>>>
>>>>>
>>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt
>>>>> node, and then create a GlusterFS using them?
>>>>>
>>>>> 2 - Instead, should I create a JBOD array made of all server's
disks?
>>>>>
>>>>> 3 - What is the best Gluster configuration to provide for HA
>>>>> while not consuming too much disk space?
>>>>>
>>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and
>>>>> the virtualization environment, benefits from tiering when using
>>>>> a SSD disk? And yes, will Gluster do it by default or I have to
>>>>> configure it to do so?
>>>>>
>>>>>
>>>>> At the bottom line, what is the good practice for using GlusterFS
>>>>> in small pods for enterprises?
>>>>>
>>>>>
>>>>> You opinion/feedback will be really appreciated!
>>>>>
>>>>> Moacir
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mailerekle.magradze(a)recogizer.de
Web:www.recogizer.com
Recogizer auf LinkedInhttps://www.linkedin.com/company-beta/10039182/
Folgen Sie uns auf Twitterhttps://twitter.com/recogizer
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.
--------------D415F827B0DC677345E35156
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Thanks for the detailed answer Erekle.</p>
<p>I conclude that it is worth in any scenario to have a arbiter
node in order to avoid wasting more disk space to RAID X + Gluster
Replication on the top of it. The cost seems much lower if you
consider running costs of the whole storage and compare it with
the cost to build the arbiter node. Even having a fully redundant
arbiter service with 2 nodes would make it wort on a larger
deployment.</p>
<p>Regards<br>
Fernando</p>
<div class="moz-cite-prefix">On 07/08/2017 17:07, Erekle Magradze
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b61c0164-c933-8204-a949-0aa303983548@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Fernando (sorry for misspelling your name, I used a
different keyboard),</p>
<p>So let's go with the following scenarios:</p>
<p>1. Let's say you have two servers (replication factor is 2),
i.e. two bricks per volume, in this case it is strongly
recommended to have the arbiter node, the metadata storage that
will guarantee avoiding the split brain situation, in this case
for arbiter you don't even need a disk with lots of space, it's
enough to have a tiny ssd but hosted on a separate server.
Advantage of such setup is that you don't need the RAID 1 for
each brick, you have the metadata information stored in arbiter
node and brick replacement is easy.</p>
<p>2. If you have odd number of bricks (let's say 3, i.e.
replication factor is 3) in your volume and you didn't create
the arbiter node as well as you didn't configure the quorum, in
this case the entire load for keeping the consistency of the
volume resides on all 3 servers, each of them is important and
each brick contains key information, they need to cross-check
each other (that's what people usually do with the first try of
gluster :) ), in this case replacing a brick is a big pain and
in this case RAID 1 is a good option to have (that's the
disadvantage, i.e. loosing the space and not having the JBOD
option) advantage is that you don't have the to have additional
arbiter node.</p>
<p>3. You have odd number of bricks and configured arbiter node,
in this case you can easily go with JBOD, however a good
practice would be to have a RAID 1 for arbiter disks (tiny 128GB
SSD-s ar perfectly sufficient for volumes with 10s of TB-s in
size.)</p>
<p>That's basically it</p>
<p>The rest about the reliability and setup scenarios you can find
in gluster documentation, especially look for quorum and arbiter
node configs+options.</p>
<p>Cheers</p>
<p>Erekle</p>
P.S. What I was mentioning, regarding a good practice is mostly
related to the operations of gluster not installation or
deployment, i.e. not the conceptual understanding of gluster
(conceptually it's a JBOD system).<br>
<br>
<div class="moz-cite-prefix">On 08/07/2017 05:41 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Thanks for the clarification Erekle.</p>
<p>However I get surprised with this way of operating from
GlusterFS as it adds another layer of complexity to the system
(either a hardware or software RAID) before the gluster config
and increase the system's overall costs.<br>
</p>
<p>An important point to consider is: In RAID configuration you
already have space 'wasted' in order to build redundancy
(either RAID 1, 5, or 6). Then when you have GlusterFS on the
top of several RAIDs you have again more data replicated so
you end up with the same data consuming more space in a group
of disks and again on the top of several RAIDs depending on
the Gluster configuration you have (in a RAID 1 config the
same data is replicated 4 times).</p>
<p>Yet another downside of having a RAID (specially RAID 5 or 6)
is that it reduces considerably the write speeds as each group
of disks will end up having the write speed of a single disk
as all other disks of that group have to wait for each other
to write as well.<br>
</p>
<p>Therefore if Gluster already replicates data why does it
create this big pain you mentioned if the data is replicated
somewhere else, can still be retrieved to both serve clients
and reconstruct the equivalent disk when it is replaced ?</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 10:26, Erekle
Magradze wrote:<br>
</div>
<blockquote type="cite"
cite="mid:aa829d07-fa77-3ed9-2500-e33cc01414b6@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Frenando,</p>
<p>Here is my experience, if you consider a particular hard
drive as a brick for gluster volume and it dies, i.e. it
becomes not accessible it's a huge hassle to discard that
brick and exchange with another one, since gluster some
tries to access that broken brick and it's causing (at least
it cause for me) a big pain, therefore it's better to have a
RAID as brick, i.e. have RAID 1 (mirroring) for each brick,
in this case if the disk is down you can easily exchange it
and rebuild the RAID without going offline, i.e switching
off the volume doing brick manipulations and switching it
back on.<br>
</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 03:04 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>For any RAID 5 or 6 configuration I normally follow a
simple gold rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the
recommendation to use any RAID with GlusterFS. I always
thought that GlusteFS likes to work in JBOD mode and
control the disks (bricks) directlly so you can create
whatever distribution rule you wish, and if a single disk
fails you just replace it and which obviously have the
data replicated from another. The only downside of using
in this way is that the replication data will be flow
accross all servers but that is not much a big issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and
JBOD + GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin
Acosta wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono">I have recently installed multiple Red Hat
Virtualization hosts for several different companies,
and have dealt with the Red Hat Support Team in depth
about optimal configuration in regards to setting up
GlusterFS most efficiently and I wanted to share with
you what I learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono">In general Red Hat Virtualization team frowns
upon using each DISK of the system as just a JBOD,
sure there is some protection by having the data
replicated, however, the recommendation is to use RAID
6 (preferred) or RAID-5, or at least RAID-1 at the
very least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono">Here is the direct quote from Red
Hat when I asked about RAID and Bricks:</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>"A typical Gluster
configuration
would use RAID underneath the bricks. RAID 6 is most
typical as it gives you 2 disk failure protection,
but RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired
replication on top of that. The most popular way of
doing this would be distributed replicated with 2x
replication. In general you'll get better
performance with larger bricks. 12 drives is often a
sweet spot. Another option would be to create a
separate tier using all SSD’s.” </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><br>
</div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>In order to SSD tiering from my
understanding you would need 1 x NVMe drive in each
server, or 4 x SSD hot tier (it needs to be
distributed, replicated for the hot tier if not
using NVME). So with you only having 1 SSD drive in
each server, I’d suggest maybe looking into the NVME
option. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Since your using only
3-servers,
what I’d probably suggest is to do (2 Replicas +
Arbiter Node), this setup actually doesn’t require
the 3rd server to have big drives at all as it only
stores meta-data about the files and not actually a
full copy. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Please see the attached
document
that was given to me by Red Hat to get more
information on this. Hope this information helps
you.</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature"
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM,
Moacir Ferreira (<a
href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite"
class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt
"pod",
made of 3 servers, each with 2 CPU sockets
of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD.
The idea is to use GlusterFS to provide HA
for the VMs. The 3 servers have a dual 40Gb
NIC and a dual 10Gb NIC. So my intention is
to create a loop like a server triangle
using the 40Gb NICs for virtualization files
(VMs .qcow2) access and to move VMs around
the pod (east /west traffic) while using the
10Gb interfaces for giving services to the
outside world (north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I
deploy GlusterFS in such oVirt scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5),
one on each oVirt node, and then create a
GlusterFS using them?</p>
<p>2 - Instead, should I create a JBOD array
made of all server's disks?</p>
<p>3 - What is the best Gluster configuration to
provide for HA while not consuming too much
disk space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am
planning to build, and the virtualization
environment, benefits from tiering when using
a SSD disk? And yes, will Gluster do it by
default or I have to configure it to do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice
for using GlusterFS in small pods for
enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really
appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre
wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail <a class="moz-txt-link-abbreviated"
href="mailto:erekle.magradze@recogizer.de"
moz-do-not-send="true">erekle.magradze(a)recogizer.de</a>
Web: <a class="moz-txt-link-abbreviated"
href="http://www.recogizer.com"
moz-do-not-send="true">www.recogizer.com</a>
Recogizer auf LinkedIn <a class="moz-txt-link-freetext"
href="https://www.linkedin.com/company-beta/10039182/"
moz-do-not-send="true">https://www.linkedin.com/company-beta...
Folgen Sie uns auf Twitter <a class="moz-txt-link-freetext"
href="https://twitter.com/recogizer"
moz-do-not-send="true">https://twitter.com/recogizer</a>
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.</pre>
</blockquote>
<br>
</body>
</html>
--------------D415F827B0DC677345E35156--
12:11 a.m.
This is a multi-part message in MIME format.
--------------16E98D99A2429E767D89172D
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi Fernando,
Indeed, having and arbiter node is always a good idea, and it saves
costs a lot.
Good luck with your setup.
Cheers
Erekle
On 07.08.2017 23:03, FERNANDO FREDIANI wrote:
Thanks for the detailed answer Erekle.
I conclude that it is worth in any scenario to have a arbiter node in
order to avoid wasting more disk space to RAID X + Gluster Replication
on the top of it. The cost seems much lower if you consider running
costs of the whole storage and compare it with the cost to build the
arbiter node. Even having a fully redundant arbiter service with 2
nodes would make it wort on a larger deployment.
Regards
Fernando
On 07/08/2017 17:07, Erekle Magradze wrote:
>
> Hi Fernando (sorry for misspelling your name, I used a different
> keyboard),
>
> So let's go with the following scenarios:
>
> 1. Let's say you have two servers (replication factor is 2), i.e. two
> bricks per volume, in this case it is strongly recommended to have
> the arbiter node, the metadata storage that will guarantee avoiding
> the split brain situation, in this case for arbiter you don't even
> need a disk with lots of space, it's enough to have a tiny ssd but
> hosted on a separate server. Advantage of such setup is that you
> don't need the RAID 1 for each brick, you have the metadata
> information stored in arbiter node and brick replacement is easy.
>
> 2. If you have odd number of bricks (let's say 3, i.e. replication
> factor is 3) in your volume and you didn't create the arbiter node as
> well as you didn't configure the quorum, in this case the entire load
> for keeping the consistency of the volume resides on all 3 servers,
> each of them is important and each brick contains key information,
> they need to cross-check each other (that's what people usually do
> with the first try of gluster :) ), in this case replacing a brick is
> a big pain and in this case RAID 1 is a good option to have (that's
> the disadvantage, i.e. loosing the space and not having the JBOD
> option) advantage is that you don't have the to have additional
> arbiter node.
>
> 3. You have odd number of bricks and configured arbiter node, in this
> case you can easily go with JBOD, however a good practice would be to
> have a RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly
> sufficient for volumes with 10s of TB-s in size.)
>
> That's basically it
>
> The rest about the reliability and setup scenarios you can find in
> gluster documentation, especially look for quorum and arbiter node
> configs+options.
>
> Cheers
>
> Erekle
>
> P.S. What I was mentioning, regarding a good practice is mostly
> related to the operations of gluster not installation or deployment,
> i.e. not the conceptual understanding of gluster (conceptually it's a
> JBOD system).
>
> On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:
>>
>> Thanks for the clarification Erekle.
>>
>> However I get surprised with this way of operating from GlusterFS as
>> it adds another layer of complexity to the system (either a hardware
>> or software RAID) before the gluster config and increase the
>> system's overall costs.
>>
>> An important point to consider is: In RAID configuration you already
>> have space 'wasted' in order to build redundancy (either RAID 1, 5,
>> or 6). Then when you have GlusterFS on the top of several RAIDs you
>> have again more data replicated so you end up with the same data
>> consuming more space in a group of disks and again on the top of
>> several RAIDs depending on the Gluster configuration you have (in a
>> RAID 1 config the same data is replicated 4 times).
>>
>> Yet another downside of having a RAID (specially RAID 5 or 6) is
>> that it reduces considerably the write speeds as each group of disks
>> will end up having the write speed of a single disk as all other
>> disks of that group have to wait for each other to write as well.
>>
>> Therefore if Gluster already replicates data why does it create this
>> big pain you mentioned if the data is replicated somewhere else, can
>> still be retrieved to both serve clients and reconstruct the
>> equivalent disk when it is replaced ?
>>
>> Fernando
>>
>>
>> On 07/08/2017 10:26, Erekle Magradze wrote:
>>>
>>> Hi Frenando,
>>>
>>> Here is my experience, if you consider a particular hard drive as a
>>> brick for gluster volume and it dies, i.e. it becomes not
>>> accessible it's a huge hassle to discard that brick and exchange
>>> with another one, since gluster some tries to access that broken
>>> brick and it's causing (at least it cause for me) a big pain,
>>> therefore it's better to have a RAID as brick, i.e. have RAID 1
>>> (mirroring) for each brick, in this case if the disk is down you
>>> can easily exchange it and rebuild the RAID without going offline,
>>> i.e switching off the volume doing brick manipulations and
>>> switching it back on.
>>>
>>> Cheers
>>>
>>> Erekle
>>>
>>>
>>> On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
>>>>
>>>> For any RAID 5 or 6 configuration I normally follow a simple gold
>>>> rule which gave good results so far:
>>>> - up to 4 disks RAID 5
>>>> - 5 or more disks RAID 6
>>>>
>>>> However I didn't really understand well the recommendation to use
>>>> any RAID with GlusterFS. I always thought that GlusteFS likes to
>>>> work in JBOD mode and control the disks (bricks) directlly so you
>>>> can create whatever distribution rule you wish, and if a single
>>>> disk fails you just replace it and which obviously have the data
>>>> replicated from another. The only downside of using in this way is
>>>> that the replication data will be flow accross all servers but
>>>> that is not much a big issue.
>>>>
>>>> Anyone can elaborate about Using RAID + GlusterFS and JBOD +
>>>> GlusterFS.
>>>>
>>>> Thanks
>>>> Regards
>>>> Fernando
>>>>
>>>>
>>>> On 07/08/2017 03:46, Devin Acosta wrote:
>>>>>
>>>>> Moacir,
>>>>>
>>>>> I have recently installed multiple Red Hat Virtualization hosts
>>>>> for several different companies, and have dealt with the Red Hat
>>>>> Support Team in depth about optimal configuration in regards to
>>>>> setting up GlusterFS most efficiently and I wanted to share with
>>>>> you what I learned.
>>>>>
>>>>> In general Red Hat Virtualization team frowns upon using each
>>>>> DISK of the system as just a JBOD, sure there is some protection
>>>>> by having the data replicated, however, the recommendation is to
>>>>> use RAID 6 (preferred) or RAID-5, or at least RAID-1 at the very
>>>>> least.
>>>>>
>>>>> Here is the direct quote from Red Hat when I asked about RAID and
>>>>> Bricks:
>>>>> /
>>>>> /
>>>>> /"A typical Gluster configuration would use RAID underneath the
>>>>> bricks. RAID 6 is most typical as it gives you 2 disk failure
>>>>> protection, but RAID 5 could be used too. Once you have the
>>>>> RAIDed bricks, you'd then apply the desired replication on top of
>>>>> that. The most popular way of doing this would be distributed
>>>>> replicated with 2x replication. In general you'll get better
>>>>> performance with larger bricks. 12 drives is often a sweet spot.
>>>>> Another option would be to create a separate tier using all SSD’s.”
/
>>>>>
>>>>> /In order to SSD tiering from my understanding you would need 1 x
>>>>> NVMe drive in each server, or 4 x SSD hot tier (it needs to be
>>>>> distributed, replicated for the hot tier if not using NVME). So
>>>>> with you only having 1 SSD drive in each server, I’d suggest
>>>>> maybe looking into the NVME option. /
>>>>> /
>>>>> /
>>>>> /Since your using only 3-servers, what I’d probably suggest is to
>>>>> do (2 Replicas + Arbiter Node), this setup actually doesn’t
>>>>> require the 3rd server to have big drives at all as it only
>>>>> stores meta-data about the files and not actually a full copy. /
>>>>> /
>>>>> /
>>>>> /Please see the attached document that was given to me by Red Hat
>>>>> to get more information on this. Hope this information helps you./
>>>>> /
>>>>> /
>>>>>
>>>>> --
>>>>>
>>>>> Devin Acosta, RHCA, RHVCA
>>>>> Red Hat Certified Architect
>>>>>
>>>>> On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
>>>>> (moacirferreira(a)hotmail.com
<mailto:moacirferreira@hotmail.com>)
>>>>> wrote:
>>>>>
>>>>>> I am willing to assemble a oVirt "pod", made of 3
servers, each
>>>>>> with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The
>>>>>> idea is to use GlusterFS to provide HA for the VMs. The 3
>>>>>> servers have a dual 40Gb NIC and a dual 10Gb NIC. So my
>>>>>> intention is to create a loop like a server triangle using the
>>>>>> 40Gb NICs for virtualization files (VMs .qcow2) access and to
>>>>>> move VMs around the pod (east /west traffic) while using the
>>>>>> 10Gb interfaces for giving services to the outside world
>>>>>> (north/south traffic).
>>>>>>
>>>>>>
>>>>>> This said, my first question is: How should I deploy GlusterFS
>>>>>> in such oVirt scenario? My questions are:
>>>>>>
>>>>>>
>>>>>> 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt
>>>>>> node, and then create a GlusterFS using them?
>>>>>>
>>>>>> 2 - Instead, should I create a JBOD array made of all
server's
>>>>>> disks?
>>>>>>
>>>>>> 3 - What is the best Gluster configuration to provide for HA
>>>>>> while not consuming too much disk space?
>>>>>>
>>>>>> 4 - Does a oVirt hypervisor pod like I am planning to build, and
>>>>>> the virtualization environment, benefits from tiering when using
>>>>>> a SSD disk? And yes, will Gluster do it by default or I have to
>>>>>> configure it to do so?
>>>>>>
>>>>>>
>>>>>> At the bottom line, what is the good practice for using
>>>>>> GlusterFS in small pods for enterprises?
>>>>>>
>>>>>>
>>>>>> You opinion/feedback will be really appreciated!
>>>>>>
>>>>>> Moacir
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>
> --
> Recogizer Group GmbH
>
> Dr.rer.nat. Erekle Magradze
> Lead Big Data Engineering & DevOps
> Rheinwerkallee 2, 53227 Bonn
> Tel: +49 228 29974555
>
> E-Mailerekle.magradze(a)recogizer.de
> Web:www.recogizer.com
>
> Recogizer auf LinkedInhttps://www.linkedin.com/company-beta/10039182/
> Folgen Sie uns auf Twitterhttps://twitter.com/recogizer
>
> -----------------------------------------------------------------
> Recogizer Group GmbH
> Geschäftsführer: Oliver Habisch, Carsten Kreutze
> Handelsregister: Amtsgericht Bonn HRB 20724
> Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
>
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten
haben,
> informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.
--------------16E98D99A2429E767D89172D
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Fernando,<br>
</p>
<p>Indeed, having and arbiter node is always a good idea, and it
saves costs a lot.</p>
<p>Good luck with your setup.</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 07.08.2017 23:03, FERNANDO FREDIANI
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:47b0f3b5-a836-d5c2-7cf4-c0147aa3948f@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Thanks for the detailed answer Erekle.</p>
<p>I conclude that it is worth in any scenario to have a arbiter
node in order to avoid wasting more disk space to RAID X +
Gluster Replication on the top of it. The cost seems much lower
if you consider running costs of the whole storage and compare
it with the cost to build the arbiter node. Even having a fully
redundant arbiter service with 2 nodes would make it wort on a
larger deployment.</p>
<p>Regards<br>
Fernando</p>
<div class="moz-cite-prefix">On 07/08/2017 17:07, Erekle Magradze
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b61c0164-c933-8204-a949-0aa303983548@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Fernando (sorry for misspelling your name, I used a
different keyboard),</p>
<p>So let's go with the following scenarios:</p>
<p>1. Let's say you have two servers (replication factor is 2),
i.e. two bricks per volume, in this case it is strongly
recommended to have the arbiter node, the metadata storage
that will guarantee avoiding the split brain situation, in
this case for arbiter you don't even need a disk with lots of
space, it's enough to have a tiny ssd but hosted on a separate
server. Advantage of such setup is that you don't need the
RAID 1 for each brick, you have the metadata information
stored in arbiter node and brick replacement is easy.</p>
<p>2. If you have odd number of bricks (let's say 3, i.e.
replication factor is 3) in your volume and you didn't create
the arbiter node as well as you didn't configure the quorum,
in this case the entire load for keeping the consistency of
the volume resides on all 3 servers, each of them is important
and each brick contains key information, they need to
cross-check each other (that's what people usually do with the
first try of gluster :) ), in this case replacing a brick is a
big pain and in this case RAID 1 is a good option to have
(that's the disadvantage, i.e. loosing the space and not
having the JBOD option) advantage is that you don't have the
to have additional arbiter node.</p>
<p>3. You have odd number of bricks and configured arbiter node,
in this case you can easily go with JBOD, however a good
practice would be to have a RAID 1 for arbiter disks (tiny
128GB SSD-s ar perfectly sufficient for volumes with 10s of
TB-s in size.)</p>
<p>That's basically it</p>
<p>The rest about the reliability and setup scenarios you can
find in gluster documentation, especially look for quorum and
arbiter node configs+options.</p>
<p>Cheers</p>
<p>Erekle</p>
P.S. What I was mentioning, regarding a good practice is mostly
related to the operations of gluster not installation or
deployment, i.e. not the conceptual understanding of gluster
(conceptually it's a JBOD system).<br>
<br>
<div class="moz-cite-prefix">On 08/07/2017 05:41 PM, FERNANDO
FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:c7a1c2e1-57c3-9fa5-0710-ebee3f3fa069@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Thanks for the clarification Erekle.</p>
<p>However I get surprised with this way of operating from
GlusterFS as it adds another layer of complexity to the
system (either a hardware or software RAID) before the
gluster config and increase the system's overall costs.<br>
</p>
<p>An important point to consider is: In RAID configuration
you already have space 'wasted' in order to build redundancy
(either RAID 1, 5, or 6). Then when you have GlusterFS on
the top of several RAIDs you have again more data replicated
so you end up with the same data consuming more space in a
group of disks and again on the top of several RAIDs
depending on the Gluster configuration you have (in a RAID 1
config the same data is replicated 4 times).</p>
<p>Yet another downside of having a RAID (specially RAID 5 or
6) is that it reduces considerably the write speeds as each
group of disks will end up having the write speed of a
single disk as all other disks of that group have to wait
for each other to write as well.<br>
</p>
<p>Therefore if Gluster already replicates data why does it
create this big pain you mentioned if the data is replicated
somewhere else, can still be retrieved to both serve clients
and reconstruct the equivalent disk when it is replaced ?</p>
<p>Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 10:26, Erekle
Magradze wrote:<br>
</div>
<blockquote type="cite"
cite="mid:aa829d07-fa77-3ed9-2500-e33cc01414b6@recogizer.de">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>Hi Frenando,</p>
<p>Here is my experience, if you consider a particular hard
drive as a brick for gluster volume and it dies, i.e. it
becomes not accessible it's a huge hassle to discard that
brick and exchange with another one, since gluster some
tries to access that broken brick and it's causing (at
least it cause for me) a big pain, therefore it's better
to have a RAID as brick, i.e. have RAID 1 (mirroring) for
each brick, in this case if the disk is down you can
easily exchange it and rebuild the RAID without going
offline, i.e switching off the volume doing brick
manipulations and switching it back on.<br>
</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/07/2017 03:04 PM,
FERNANDO FREDIANI wrote:<br>
</div>
<blockquote type="cite"
cite="mid:63bac47b-afe6-0258-d3d7-e545a5004c30@upx.com">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<p>For any RAID 5 or 6 configuration I normally follow a
simple gold rule which gave good results so far:<br>
- up to 4 disks RAID 5<br>
- 5 or more disks RAID 6</p>
<p>However I didn't really understand well the
recommendation to use any RAID with GlusterFS. I always
thought that GlusteFS likes to work in JBOD mode and
control the disks (bricks) directlly so you can create
whatever distribution rule you wish, and if a single
disk fails you just replace it and which obviously have
the data replicated from another. The only downside of
using in this way is that the replication data will be
flow accross all servers but that is not much a big
issue.</p>
<p>Anyone can elaborate about Using RAID + GlusterFS and
JBOD + GlusterFS.</p>
<p>Thanks<br>
Regards<br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/08/2017 03:46, Devin
Acosta wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANCGKEp4XGs0U+Qs78eEmqCNtvpLY-Azjb5DcGhZ9yiKTBEEfw@mail.gmail.com">
<style>body{font-family:Helvetica,Arial;font-size:13px}</style>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono">Moacir,</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono">I have recently installed multiple Red Hat
Virtualization hosts for several different
companies, and have dealt with the Red Hat Support
Team in depth about optimal configuration in regards
to setting up GlusterFS most efficiently and I
wanted to share with you what I learned.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono">In general Red Hat Virtualization team frowns
upon using each DISK of the system as just a JBOD,
sure there is some protection by having the data
replicated, however, the recommendation is to use
RAID 6 (preferred) or RAID-5, or at least RAID-1 at
the very least.</font></div>
<div id="bloop_customfont"
style="color:rgb(0,0,0);margin:0px"><font
face="Input
Mono"><br>
</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono">Here is the direct quote from Red
Hat when I asked about RAID and Bricks:</font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>"A typical Gluster
configuration would use RAID underneath the
bricks. RAID 6 is most typical as it gives you 2
disk failure protection, but RAID 5 could be used
too. Once you have the RAIDed bricks, you'd then
apply the desired replication on top of that. The
most popular way of doing this would be
distributed replicated with 2x replication. In
general you'll get better performance with larger
bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using
all SSD’s.” </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><br>
</div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>In order to SSD tiering from
my
understanding you would need 1 x NVMe drive in
each server, or 4 x SSD hot tier (it needs to be
distributed, replicated for the hot tier if not
using NVME). So with you only having 1 SSD drive
in each server, I’d suggest maybe looking into the
NVME option. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Since your using only
3-servers, what I’d probably suggest is to do (2
Replicas + Arbiter Node), this setup actually
doesn’t require the 3rd server to have big drives
at all as it only stores meta-data about the files
and not actually a full copy. </i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i>Please see the attached
document that was given to me by Red Hat to get
more information on this. Hope this information
helps you.</i></font></div>
<div id="bloop_customfont"
style="margin:0px"><font
face="Input Mono"><i><br>
</i></font></div>
<br>
<div id="bloop_sign_1502087376725469184"
class="bloop_sign"><span
style="font-family:'helvetica
Neue',helvetica;font-size:14px">--</span><br
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div class="gmail_signature"
style="font-family:'helvetica
Neue',helvetica;font-size:14px">
<div dir="ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
</div>
</div>
</div>
<br>
<p class="airmail_on">On August 6, 2017 at 7:29:29 PM,
Moacir Ferreira (<a
href="mailto:moacirferreira@hotmail.com"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>)
wrote:</p>
<blockquote type="cite"
class="clean_bq"><span>
<div dir="ltr">
<div>
<title></title>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif"
dir="ltr">
<p><span>I am willing to assemble a oVirt
"pod", made of 3 servers, each with 2 CPU
sockets of 12 cores, 256GB RAM, 7 HDD 10K,
1 SSD. The idea is to use GlusterFS to
provide HA for the VMs. The 3 servers have
a dual 40Gb NIC and a dual 10Gb NIC. So my
intention is to create a loop like a
server triangle using the 40Gb NICs for
virtualization files (VMs .qcow2) access
and to move VMs around the pod (east /west
traffic) while using the 10Gb interfaces
for giving services to the outside world
(north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should
I deploy GlusterFS in such oVirt scenario?
My questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5),
one on each oVirt node, and then create a
GlusterFS using them?</p>
<p>2 - Instead, should I create a JBOD array
made of all server's disks?</p>
<p>3 - What is the best Gluster configuration
to provide for HA while not consuming too
much disk space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am
planning to build, and the virtualization
environment, benefits from tiering when
using a SSD disk? And yes, will Gluster do
it by default or I have to configure it to
do so?</p>
<p><br>
</p>
<p>At the bottom line, what is the good
practice for using GlusterFS in small pods
for enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really
appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________
<br>
Users mailing list <br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
</div>
</div>
</span></blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre
wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre
wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail <a class="moz-txt-link-abbreviated"
href="mailto:erekle.magradze@recogizer.de"
moz-do-not-send="true">erekle.magradze(a)recogizer.de</a>
Web: <a class="moz-txt-link-abbreviated"
href="http://www.recogizer.com"
moz-do-not-send="true">www.recogizer.com</a>
Recogizer auf LinkedIn <a class="moz-txt-link-freetext"
href="https://www.linkedin.com/company-beta/10039182/"
moz-do-not-send="true">https://www.linkedin.com/company-beta...
Folgen Sie uns auf Twitter <a class="moz-txt-link-freetext"
href="https://twitter.com/recogizer"
moz-do-not-send="true">https://twitter.com/recogizer</a>
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.</pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>
--------------16E98D99A2429E767D89172D--
9:29 a.m.
On Tue, Aug 8, 2017 at 12:03 AM, FERNANDO FREDIANI <
fernando.frediani(a)upx.com> wrote:
Thanks for the detailed answer Erekle.
I conclude that it is worth in any scenario to have a arbiter node in
order to avoid wasting more disk space to RAID X + Gluster Replication on
the top of it. The cost seems much lower if you consider running costs of
the whole storage and compare it with the cost to build the arbiter node.
Even having a fully redundant arbiter service with 2 nodes would make it
wort on a larger deployment.
Note that although you get the same consistency as a replica 3 setup, a
2+arbiter gives you data availability as a replica 2 setup. May or may not
be OK with your high availability requirements.
Y.
Regards
Fernando
On 07/08/2017 17:07, Erekle Magradze wrote:
Hi Fernando (sorry for misspelling your name, I used a different keyboard),
So let's go with the following scenarios:
1. Let's say you have two servers (replication factor is 2), i.e. two
bricks per volume, in this case it is strongly recommended to have the
arbiter node, the metadata storage that will guarantee avoiding the split
brain situation, in this case for arbiter you don't even need a disk with
lots of space, it's enough to have a tiny ssd but hosted on a separate
server. Advantage of such setup is that you don't need the RAID 1 for each
brick, you have the metadata information stored in arbiter node and brick
replacement is easy.
2. If you have odd number of bricks (let's say 3, i.e. replication factor
is 3) in your volume and you didn't create the arbiter node as well as you
didn't configure the quorum, in this case the entire load for keeping the
consistency of the volume resides on all 3 servers, each of them is
important and each brick contains key information, they need to cross-check
each other (that's what people usually do with the first try of gluster :)
), in this case replacing a brick is a big pain and in this case RAID 1 is
a good option to have (that's the disadvantage, i.e. loosing the space and
not having the JBOD option) advantage is that you don't have the to have
additional arbiter node.
3. You have odd number of bricks and configured arbiter node, in this case
you can easily go with JBOD, however a good practice would be to have a
RAID 1 for arbiter disks (tiny 128GB SSD-s ar perfectly sufficient for
volumes with 10s of TB-s in size.)
That's basically it
The rest about the reliability and setup scenarios you can find in gluster
documentation, especially look for quorum and arbiter node configs+options.
Cheers
Erekle
P.S. What I was mentioning, regarding a good practice is mostly related to
the operations of gluster not installation or deployment, i.e. not the
conceptual understanding of gluster (conceptually it's a JBOD system).
On 08/07/2017 05:41 PM, FERNANDO FREDIANI wrote:
Thanks for the clarification Erekle.
However I get surprised with this way of operating from GlusterFS as it
adds another layer of complexity to the system (either a hardware or
software RAID) before the gluster config and increase the system's overall
costs.
An important point to consider is: In RAID configuration you already have
space 'wasted' in order to build redundancy (either RAID 1, 5, or 6). Then
when you have GlusterFS on the top of several RAIDs you have again more
data replicated so you end up with the same data consuming more space in a
group of disks and again on the top of several RAIDs depending on the
Gluster configuration you have (in a RAID 1 config the same data is
replicated 4 times).
Yet another downside of having a RAID (specially RAID 5 or 6) is that it
reduces considerably the write speeds as each group of disks will end up
having the write speed of a single disk as all other disks of that group
have to wait for each other to write as well.
Therefore if Gluster already replicates data why does it create this big
pain you mentioned if the data is replicated somewhere else, can still be
retrieved to both serve clients and reconstruct the equivalent disk when it
is replaced ?
Fernando
On 07/08/2017 10:26, Erekle Magradze wrote:
Hi Frenando,
Here is my experience, if you consider a particular hard drive as a brick
for gluster volume and it dies, i.e. it becomes not accessible it's a huge
hassle to discard that brick and exchange with another one, since gluster
some tries to access that broken brick and it's causing (at least it cause
for me) a big pain, therefore it's better to have a RAID as brick, i.e.
have RAID 1 (mirroring) for each brick, in this case if the disk is down
you can easily exchange it and rebuild the RAID without going offline, i.e
switching off the volume doing brick manipulations and switching it back on.
Cheers
Erekle
On 08/07/2017 03:04 PM, FERNANDO FREDIANI wrote:
For any RAID 5 or 6 configuration I normally follow a simple gold rule
which gave good results so far:
- up to 4 disks RAID 5
- 5 or more disks RAID 6
However I didn't really understand well the recommendation to use any RAID
with GlusterFS. I always thought that GlusteFS likes to work in JBOD mode
and control the disks (bricks) directlly so you can create whatever
distribution rule you wish, and if a single disk fails you just replace it
and which obviously have the data replicated from another. The only
downside of using in this way is that the replication data will be flow
accross all servers but that is not much a big issue.
Anyone can elaborate about Using RAID + GlusterFS and JBOD + GlusterFS.
Thanks
Regards
Fernando
On 07/08/2017 03:46, Devin Acosta wrote:
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for
several different companies, and have dealt with the Red Hat Support Team
in depth about optimal configuration in regards to setting up GlusterFS
most efficiently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the
system as just a JBOD, sure there is some protection by having the data
replicated, however, the recommendation is to use RAID 6 (preferred) or
RAID-5, or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
*"A typical Gluster configuration would use RAID underneath the bricks.
RAID 6 is most typical as it gives you 2 disk failure protection, but RAID
5 could be used too. Once you have the RAIDed bricks, you'd then apply the
desired replication on top of that. The most popular way of doing this
would be distributed replicated with 2x replication. In general you'll get
better performance with larger bricks. 12 drives is often a sweet spot.
Another option would be to create a separate tier using all SSD’s.” *
*In order to SSD tiering from my understanding you would need 1 x NVMe
drive in each server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME). So with you only having 1
SSD drive in each server, I’d suggest maybe looking into the NVME option. *
*Since your using only 3-servers, what I’d probably suggest is to do (2
Replicas + Arbiter Node), this setup actually doesn’t require the 3rd
server to have big drives at all as it only stores meta-data about the
files and not actually a full copy. *
*Please see the attached document that was given to me by Red Hat to get
more information on this. Hope this information helps you.*
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (
moacirferreira(a)hotmail.com) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and
a dual 10Gb NIC. So my intention is to create a loop like a server triangle
using the 40Gb NICs for virtualization files (VMs .qcow2) access and to
move VMs around the pod (east /west traffic) while using the 10Gb
interfaces for giving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
then create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555 <+49%20228%2029974555>
E-Mail erekle.magradze(a)recogizer.de
Web: www.recogizer.com
Recogizer auf LinkedIn https://www.linkedin.com/company-beta/10039182/
Folgen Sie uns auf Twitter https://twitter.com/recogizer
-----------------------------------------------------------------
Recogizer Group GmbH
Geschäftsführer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und löschen Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:29 p.m.
--_000_VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50VI1P190MB0285EURP_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi Devin,
Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5=
to assemble a single disk on each server. In this case, the SSD will not m=
ake any difference, right? I guess that to be possible to use it, the SSD s=
hould not be part of the RAID 5. In this case I could create a logical volu=
me made of the RAIDed brick and then extend it using the SSD. I.e.: Using g=
deploy:
[disktype]
jbod
....
[pv1]
action=3Dcreate
devices=3Dsdb, sdc
wipefs=3Dyes
ignore_vg_erros=3Dno
[vg1]
action=3Dcreate
vgname=3Dgluster_vg_jbod
pvname=3Dsdb
ignore_vg_erros=3Dno
[vg2]
action=3Dextend
vgname=3Dgluster_vg_jbod
pvname=3Dsdc
ignore_vg_erros=3Dno
But will Gluster be able to auto-detect and use this SSD brick for tiering?=
Do I have to do some other configurations? Also, as the VM files (.qcow2) =
are quite big will I benefit from tiering? This is wrong and my approach sh=
ould be other?
Thanks,
Moacir
________________________________
From: Devin Acosta <devin(a)pabstatencio.com Sent:
Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for several=
different companies, and have dealt with the Red Hat Support Team in depth=
about optimal configuration in regards to setting up GlusterFS most effici=
ently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the s=
ystem as just a JBOD, sure there is some protection by having the data repl=
icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,=
or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
"A typical Gluster configuration would use RAID underneath the bricks. RAID=
6 is most typical as it gives you 2 disk failure protection, but RAID 5 co=
uld be used too. Once you have the RAIDed bricks, you'd then apply the desi=
red replication on top of that. The most popular way of doing this would be=
distributed replicated with 2x replication. In general you'll get better p=
erformance with larger bricks. 12 drives is often a sweet spot. Another opt=
ion would be to create a separate tier using all SSD=92s.=94
In order to SSD tiering from my understanding you would need 1 x NVMe drive=
in each server, or 4 x SSD hot tier (it needs to be distributed, replicate=
d for the hot tier if not using NVME). So with you only having 1 SSD drive =
in each server, I=92d suggest maybe looking into the NVME option.
Since your using only 3-servers, what I=92d probably suggest is to do (2 Re=
plicas + Arbiter Node), this setup actually doesn=92t require the 3rd serve=
r to have big drives at all as it only stores meta-data about the files and=
not actually a full copy.
Please see the attached document that was given to me by Red Hat to get mor=
e information on this. Hope this information helps you.
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50VI1P190MB0285EURP_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3DWindows-1=
252" <style
type=3D"text/css" style=3D"display:none;"><!-- P
{margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Hi Devin,</p
<p><br </p
<p>Please consider that for the OS I have a RAID 1. Now, lets say I use RAI=
D 5 to assemble a single disk on each server. In this case, the SSD will no=
t make any difference, right? I guess that to be possible to use it, the SS=
D should not be part of the RAID
5. In this case I could create a logical volume made of the RAIDed brick a=
nd then extend it using the SSD. I.e.: Using gdeploy:</p
<p><br </p
<p>[disktype]</p <p>jbod</p <p>....</p
<p>[pv1]</p <p>action=3Dcreate</p <p>devices=3Dsdb, sdc</p
<p>wipefs=3Dyes</p <p></p <p>ignore_vg_erros=3Dno<br </p <p><br </p <p></p <p>[vg1]</p
<p><b>action=3Dcreate</b></p
<p>vgname=3Dgluster_vg_jbod</p
<p>pvname=3Dsdb</p
<p>ignore_vg_erros=3Dno<br
</p <p><br </p <p>[vg2]</p <p><b>action=3Dextend</b></p <p>vgname=3Dgluster_vg_jbod</p <p>pvname=3Dsdc</p
<p>ignore_vg_erros=3Dno<br
</p <br
<p>But will Gluster be able to auto-detect and use this SSD brick for tieri=
ng? Do I have to do some other configurations? Also, as the VM files (.qcow=
2) are quite big will I benefit from tiering? This is wrong and my approach=
should be other?</p <p><br </p <p>Thanks,</p <p>Moacir<br
</p <p><br </p <br <div
style=3D"color: rgb(49, 55, 57);" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Devin Acosta <de=
vin(a)pabstatencio.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 7:46 AM<br <b>To:</b> Moacir
Ferreira; users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">Moacir,</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">I have recently installed multiple Red Hat Virtualizatio=
n hosts for several different companies, and have dealt with the Red Hat Su=
pport Team in depth about optimal configuration
in regards to setting up GlusterFS most efficiently and I wanted to share =
with you what I learned.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">In general Red Hat Virtualization team frowns upon using=
each DISK of the system as just a JBOD, sure there is some protection by h=
aving the data replicated, however, the
recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 =
at the very least.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><br </div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<br <div
id=3D"bloop_sign_1502087376725469184" class=3D"bloop_sign"><span
style=
=3D"font-family:'helvetica Neue',helvetica;
font-size:14px">--</span><br st=
yle=3D"font-family:'helvetica Neue',helvetica; font-size:14px" <div class=3D"gmail_signature"
style=3D"font-family:'helvetica Neue',helvet=
ica; font-size:14px" <div dir=3D"ltr" <div><br
</div <div>Devin Acosta, RHCA,
RHVCA</div <div>Red Hat Certified
Architect</div <div></div </div
</div </div <br <p
class=3D"airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (<=
a
href=3D"mailto:moacirferreira@hotmail.com">moacirferreira@hotmail.com</a>=
) wrote:</p <blockquote
type=3D"cite" class=3D"clean_bq"><span <div dir=3D"ltr"
<div></div <div <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p><span>I am willing to assemble a oVirt "pod", made
of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br </p
<p>This said, my first question is: How should I deploy GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <p>Moacir<br
</p </div _______________________________________________ <br Users mailing list <br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a> <br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
t.org/mailman/listinfo/users</a <br
</div </div </span></blockquote
</div </div </div
</body </html
--_000_VI1P190MB0285AF47BE9B7FEE3EEB5067C8B50VI1P190MB0285EURP_--
Here is the direct quote from Red Hat when I asked about RAID
and Bri=
cks:</font></div<i><br>
</i></font></div<i>"A typical Gluster configuration would use RAID
underneath the bri=
cks. RAID 6 is most typical as it gives you 2 disk failure
protection, but =
RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired replication on top of that. Th=
e most popular way of doing this would be distributed replicated with 2x re=
plication. In general you'll get better performance with larger bricks=
. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all SSD=92s.=94 </i><=
/font></div<i>In order to SSD tiering from my understanding you
would need 1 x N=
VMe drive in each server, or 4 x SSD hot tier (it needs to be
distributed, =
replicated for the hot tier if not using NVME).
So with you only having 1 SSD drive in each server, I=92d suggest may=
be looking into the NVME option. </i></font></div<i><br>
</i></font></div<i>Since your using only 3-servers, what I=92d probably suggest
is to do (=
2 Replicas + Arbiter Node), this setup actually doesn=92t
require the 3=
rd server to have big drives at all as it only
stores meta-data about the files and not actually a full copy. </i></=
font></div<i><br>
</i></font></div<i>Please see the attached document that was given to me by Red
Hat to get=
more information on this. Hope this information helps
you.</i></font></div=
<i><br>
</i></font></div
9:03 a.m.
This is a multi-part message in MIME format.
------------15dc07141763cbc27688fec2dc
Content-Type: text/plain; format=flowed; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
You attach the ssd as a hot tier with a gluster command. I don't think th=
at=20
gdeploy or ovirt gui can do it.
The gluster docs and redhat docs explains tiering quite good.
/Johan
On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com> w=
rote:
Hi Devin,
Please consider that for the OS I have a RAID 1. Now, lets say I use RA=
ID 5=20
to assemble a single disk on each server. In this case, the SSD will
no=
t=20
make any difference, right? I guess that to be possible to use it,
the =
SSD=20
should not be part of the RAID 5. In this case I could create a
logical=
=20
volume made of the RAIDed brick and then extend it using the SSD.
I.e.:=
=20
Using gdeploy:
[disktype]
jbod
....
[pv1]
action=3Dcreate
devices=3Dsdb, sdc
wipefs=3Dyes
ignore_vg_erros=3Dno
[vg1]
action=3Dcreate
vgname=3Dgluster_vg_jbod
pvname=3Dsdb
ignore_vg_erros=3Dno
[vg2]
action=3Dextend
vgname=3Dgluster_vg_jbod
pvname=3Dsdc
ignore_vg_erros=3Dno
But will Gluster be able to auto-detect and use this SSD brick for tier=
ing?=20
Do I have to do some other configurations? Also, as the VM files
(.qcow=
2)=20
are quite big will I benefit from tiering? This is wrong and my
approac=
h=20
should be other?
Thanks,
Moacir
________________________________
From: Devin Acosta <devin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for sev=
eral=20
different companies, and have dealt with the Red Hat Support Team in
de=
pth=20
about optimal configuration in regards to setting up GlusterFS
most=20
efficiently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of t=
he=20
system as just a JBOD, sure there is some protection by having the
data=
=20
replicated, however, the recommendation is to use RAID 6 (preferred)
or=
=20
RAID-5, or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Brick=
s:
"A typical Gluster configuration would use RAID underneath the bricks. =
RAID=20
6 is most typical as it gives you 2 disk failure protection, but RAID
5=
=20
could be used too. Once you have the RAIDed bricks, you'd then
apply th=
e=20
desired replication on top of that. The most popular way of doing
this=20
would be distributed replicated with 2x replication. In general you'll =
get=20
better performance with larger bricks. 12 drives is often a sweet
spot.=
=20
Another option would be to create a separate tier using all
SSD=E2=80=99=
s.=E2=80=9D
In order to SSD tiering from my understanding you would need 1 x NVMe d=
rive=20
in each server, or 4 x SSD hot tier (it needs to be distributed,
replic=
ated=20
for the hot tier if not using NVME). So with you only having 1 SSD
driv=
e in=20
each server, I=E2=80=99d suggest maybe looking into the NVME option.
Since your using only 3-servers, what I=E2=80=99d probably suggest is t=
o do
(2=20
Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require
t=
he 3rd=20
server to have big drives at all as it only stores meta-data about
the=20
files and not actually a full copy.
Please see the attached document that was given to me by Red Hat to get=
=20
more information on this. Hope this information helps you.
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira=20
(moacirferreira@hotmail.com<mailto:moacirferreira@hotmail.com>) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 =
CPU=20
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to
use=20
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC=
and=20
a dual 10Gb NIC. So my intention is to create a loop like a server
tria=
ngle=20
using the 40Gb NICs for virtualization files (VMs .qcow2) access and
to=
=20
move VMs around the pod (east /west traffic) while using the 10Gb=20
interfaces for giving services to the outside world (north/south traffi=
c).
This said, my first question is: How should I deploy GlusterFS in such=20
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and =
then=20
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not=20
consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the=20
virtualization environment, benefits from tiering when using a SSD disk=
?=20
And yes, will Gluster do it by default or I have to configure it to
do =
so?
At the bottom line, what is the good practice for using GlusterFS in sm=
all=20
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
----------
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
------------15dc07141763cbc27688fec2dc
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<style type=3D"text/css" style=3D"display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
</head>
<body>
<div style=3D"color: black;">
<div style=3D"color: black;">
<p style=3D"margin: 0 0 1em 0; color: black;">You attach the ssd as a hot
tier with a gluster command. I don't think that gdeploy or ovirt gui can =
do
it.</p>
<p style=3D"margin: 0 0 1em 0; color: black;">The gluster docs and redhat
docs explains tiering quite good.</p>
<p style=3D"margin: 0 0 1em 0; color: black;">/Johan</p>
</div>
<div style=3D"color: black;">
<p
style=3D"color: black; font-size: 10pt; font-family: Arial, sans-serif; m=
argin: 10pt 0;">On
August 8, 2017 07:06:42 Moacir Ferreira &lt;moacirferreira(a)hotmail.com&gt=
;
wrote:</p>
<blockquote type=3D"cite" class=3D"gmail_quote"
style=3D"margin: 0 0 0 0.75ex; border-left: 1px solid #808080; padding-le=
ft: 0.75ex;">
<div id=3D"divtagdefaultwrapper"
style=3D"font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-=
serif;"
dir=3D"ltr">
<p>Hi Devin,</p>
<p><br>
</p>
<p>Please consider that for the OS I have a RAID 1. Now, lets say I use
RAID 5 to assemble a single disk on each server. In this case, the SSD wi=
ll
not make any difference, right? I guess that to be possible to use it, th=
e
SSD should not be part of the RAID
5. In this case I could create a logical volume made of the RAIDed brick
and then extend it using the SSD. I.e.: Using gdeploy:</p>
<p><br>
</p>
<p>[disktype]</p>
<p>jbod</p>
<p>....</p>
<p>[pv1]</p>
<p>action=3Dcreate</p>
<p>devices=3Dsdb, sdc</p>
<p>wipefs=3Dyes</p>
<p></p>
<p>ignore_vg_erros=3Dno<br>
</p>
<p><br>
</p>
<p></p>
<p>[vg1]</p>
<p><b>action=3Dcreate</b></p>
<p>vgname=3Dgluster_vg_jbod</p>
<p>pvname=3Dsdb</p>
<p>ignore_vg_erros=3Dno<br>
</p>
<p><br>
</p>
<p>[vg2]</p>
<p><b>action=3Dextend</b></p>
<p>vgname=3Dgluster_vg_jbod</p>
<p>pvname=3Dsdc</p>
<p>ignore_vg_erros=3Dno<br>
</p>
<br>
<p>But will Gluster be able to auto-detect and use this SSD brick for
tiering? Do I have to do some other configurations? Also, as the VM files
(.qcow2) are quite big will I benefit from tiering? This is wrong and my
approach should be other?</p>
<p><br>
</p>
<p>Thanks,</p>
<p>Moacir<br>
</p>
<p><br>
</p>
<br>
<div style=3D"color: rgb(49, 55, 57);">
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt"
face=3D"Calibri, sans-serif"
color=3D"#000000"><b>From:</b> Devin Acosta
&lt;devin(a)pabstatencio.com&gt;<br>
<b>Sent:</b> Monday, August 7, 2017 7:46 AM<br>
<b>To:</b> Moacir Ferreira; users(a)ovirt.org<br>
<b>Subject:</b> Re: [ovirt-users] Good practices</font>
<div> </div>
</div>
<div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono">Moacir,</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono">I have recently installed multiple Red Hat Virtualiza=
tion
hosts for several different companies, and have dealt with the Red Hat
Support Team in depth about optimal configuration
in regards to setting up GlusterFS most efficiently and I wanted to shar=
e
with you what I learned.</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono">In general Red Hat Virtualization team frowns upon us=
ing
each DISK of the system as just a JBOD, sure there is some protection by
having the data replicated, however, the
recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-=
1
at the very least.</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o">Here
is the direct quote from Red Hat when I asked about RAID and
Bricks:</font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"><i>"A typical Gluster configuration would
use RA=
ID
underneath the bricks. RAID 6 is most typical as it gives you 2 disk
failure protection, but RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired replication on top of that.
The most popular way of doing this would be distributed replicated with 2=
x
replication. In general you'll get better performance with larger
bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all
SSD=E2=80=99s.=E2=80=9D </i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><br>
</div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i>In
order to SSD tiering from my understanding you would need 1 x NVMe
drive in each server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME).
So with you only having 1 SSD drive in each server, I=E2=80=99d sug=
gest maybe
looking into the NVME option. </i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"><i>Since your using only 3-servers, what
I=E2=80=99d =
probably
suggest is to do (2 Replicas + Arbiter Node), this setup actually
doesn=E2=80=99t require the 3rd server to have big drives at all as it on=
ly
stores meta-data about the files and not actually a full
copy. </i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"><i>Please see the attached document that was given
to=
me
by Red Hat to get more information on this. Hope this information helps
you.</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<br>
<div id=3D"bloop_sign_1502087376725469184"
class=3D"bloop_sign"><span
style=3D"font-family:'helvetica Neue',helvetica;
font-size:14px">--</span=
<br
style=3D"font-family:'helvetica
Neue',helvetica; font-size:14px">
<div class=3D"gmail_signature"
style=3D"font-family:'helvetica Neue',helvetica; font-size:14px">
<div dir=3D"ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
<div></div>
</div>
</div>
</div>
<br>
<p class=3D"airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
=
(<a
href=3D"mailto:moacirferreira@hotmail.com">moacirferreira@hotmail.com</a>=
)
wrote:</p>
<blockquote type=3D"cite" class=3D"clean_bq"><span>
<div dir=3D"ltr">
<div></div>
<div>
<div id=3D"divtagdefaultwrapper" dir=3D"ltr"
style=3D"font-size:12pt; color:#000000; font-family:Calibri,Helvetica,san=
s-serif">
<p><span>I am willing to assemble a oVirt "pod", made of 3
servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD=
.
The idea is to use GlusterFS to provide HA for the VMs. The 3 servers hav=
e
a dual 40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for
virtualization files (VMs .qcow2) access and to move VMs around the pod
(east /west traffic) while using the 10Gb interfaces for giving services =
to
the outside world (north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
then create a GlusterFS using them?</p>
<p>2 - Instead, should I create a JBOD array made of all server's disks?<=
/p>
<p>3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so=
?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice for using GlusterFS in
small pods for enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a> <br>
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.org/mailman/listinfo/users</a>
<br>
</div>
</div>
</span></blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a class=3D"aqm-autolink aqm-autowrap"
href=3D"mailto:Users%40ovirt.org">Users@ovirt.org</a><br>
<a class=3D"aqm-autolink aqm-autowrap"
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.org/mailman/listinfo/users</a><br>
<br></blockquote>
</div>
</div>
</body>
</html>
------------15dc07141763cbc27688fec2dc--
1:19 p.m.
--_000_DB6P190MB0280BD0BB439D0C8A4588D00C88A0DB6P190MB0280EURP_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Thanks Johan, you brought "light" into my darkness! I went looking for the =
GlusterFS tiering how-to and it looks like quite simple to attach a SSD as =
hot tier. For those willing to read about it, go here: http://blog.gluster.=
org/2016/03/automated-tiering-in-gluster/
Now, I still have a question: VMs are made of very large .qcow2 files. My u=
nderstanding is that files in Gluster are kept all together in a single bri=
ck. If so, I will not benefit from tiering as a single SSD will not be big =
enough to fit all my large VM .qcow2 files. This would not be true if Glust=
er can store "blocks" of data that compose a large file spread on several b=
ricks. But if I am not wrong, this is one of key differences in between Glu=
sterFS and Ceph. Can you comment?
Moacir
________________________________
From: Johan Bernhardsson <johan(a)kafit.se Sent:
Tuesday, August 8, 2017 7:03 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
You attach the ssd as a hot tier with a gluster command. I don't think that=
gdeploy or ovirt gui can do it.
The gluster docs and redhat docs explains tiering quite good.
/Johan
On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com> wro=
te:
Hi Devin,
Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5=
to assemble a single disk on each server. In this case, the SSD will not m=
ake any difference, right? I guess that to be possible to use it, the SSD s=
hould not be part of the RAID 5. In this case I could create a logical volu=
me made of the RAIDed brick and then extend it using the SSD. I.e.: Using g=
deploy:
[disktype]
jbod
....
[pv1]
action=3Dcreate
devices=3Dsdb, sdc
wipefs=3Dyes
ignore_vg_erros=3Dno
[vg1]
action=3Dcreate
vgname=3Dgluster_vg_jbod
pvname=3Dsdb
ignore_vg_erros=3Dno
[vg2]
action=3Dextend
vgname=3Dgluster_vg_jbod
pvname=3Dsdc
ignore_vg_erros=3Dno
But will Gluster be able to auto-detect and use this SSD brick for tiering?=
Do I have to do some other configurations? Also, as the VM files (.qcow2) =
are quite big will I benefit from tiering? This is wrong and my approach sh=
ould be other?
Thanks,
Moacir
________________________________
From: Devin Acosta <devin(a)pabstatencio.com Sent:
Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for several=
different companies, and have dealt with the Red Hat Support Team in depth=
about optimal configuration in regards to setting up GlusterFS most effici=
ently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the s=
ystem as just a JBOD, sure there is some protection by having the data repl=
icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,=
or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
"A typical Gluster configuration would use RAID underneath the bricks. RAID=
6 is most typical as it gives you 2 disk failure protection, but RAID 5 co=
uld be used too. Once you have the RAIDed bricks, you'd then apply the desi=
red replication on top of that. The most popular way of doing this would be=
distributed replicated with 2x replication. In general you'll get better p=
erformance with larger bricks. 12 drives is often a sweet spot. Another opt=
ion would be to create a separate tier using all SSD=92s.=94
In order to SSD tiering from my understanding you would need 1 x NVMe drive=
in each server, or 4 x SSD hot tier (it needs to be distributed, replicate=
d for the hot tier if not using NVME). So with you only having 1 SSD drive =
in each server, I=92d suggest maybe looking into the NVME option.
Since your using only 3-servers, what I=92d probably suggest is to do (2 Re=
plicas + Arbiter Node), this setup actually doesn=92t require the 3rd serve=
r to have big drives at all as it only stores meta-data about the files and=
not actually a full copy.
Please see the attached document that was given to me by Red Hat to get mor=
e information on this. Hope this information helps you.
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users%40ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_DB6P190MB0280BD0BB439D0C8A4588D00C88A0DB6P190MB0280EURP_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3DWindows-1=
252" <style
type=3D"text/css" style=3D"display:none;"><!-- P
{margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Thanks Johan, you brought "light"
into my darkness! I went loo=
king for the GlusterFS tiering how-to and it looks like quite simple to att=
ach a SSD as hot tier. For those willing to read about it, go here:
<a href=3D"http://blog.gluster.org/2016/03/automated-tiering-in-gluster... c=
lass=3D"OWAAutoLink" id=3D"LPlnk939343"
previewremoved=3D"true"
http://blog.gluster.org/2016/03/automated-tiering-in-gluster/</a>&l... <p><br
</p <p>Now, I still have a
question: VMs are made of very large .qcow2 files. M=
y understanding is that files in Gluster are kept all together in a single =
brick. If so, I will not benefit from tiering as a single SSD will not be b=
ig enough to fit all my large VM
.qcow2 files. This would not be true if Gluster can store "blocks&quo=
t; of data that compose a large file spread on several bricks. But if I am =
not wrong, this is one of key differences in between GlusterFS and Ceph. Ca=
n you comment?</p <p><br </p <p>Moacir<br </p <br
<br <div style=3D"color: rgb(49,
55, 57);" <hr tabindex=3D"-1"
style=3D"display:inline-block; width:98%" <div
id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Johan Bernhardsson =
&lt;johan(a)kafit.se&gt;<br
<b>Sent:</b> Tuesday, August 8, 2017 7:03 AM<br <b>To:</b> Moacir Ferreira; Devin Acosta;
users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
style=3D"color:black" <div
style=3D"color:black" <p style=3D"margin:0 0 1em
0; color:black">You attach the ssd as a hot tier=
with a gluster command. I don't think that gdeploy or ovirt gui can do it.=
</p <p style=3D"margin:0 0 1em 0; color:black">The
gluster docs and redhat docs=
explains tiering quite good.</p <p
style=3D"margin:0 0 1em 0; color:black">/Johan</p </div <div
style=3D"color:black" <p style=3D"color:black;
font-size:10pt; font-family:Arial,sans-serif; marg=
in:10pt 0" On August 8, 2017 07:06:42 Moacir
Ferreira &lt;moacirferreira(a)hotmail.com&g=
t; wrote:</p <blockquote
type=3D"cite" class=3D"gmail_quote" style=3D"margin:0 0 0 0.75e=
x; border-left:1px solid #808080; padding-left:0.75ex" <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p>Hi Devin,</p <p><br </p <p>Please consider that for
the OS I have a RAID 1. Now, lets say I use RAI=
D 5 to assemble a single disk on each server. In this case, the SSD will no=
t make any difference, right? I guess that to be possible to use it, the SS=
D should not be part of the RAID
5. In this case I could create a logical volume made of the RAIDed brick a=
nd then extend it using the SSD. I.e.: Using gdeploy:</p
<p><br </p
<p>[disktype]</p <p>jbod</p <p>....</p
<p>[pv1]</p <p>action=3Dcreate</p <p>devices=3Dsdb, sdc</p
<p>wipefs=3Dyes</p <p></p <p>ignore_vg_erros=3Dno<br </p <p><br </p <p></p <p>[vg1]</p
<p><b>action=3Dcreate</b></p
<p>vgname=3Dgluster_vg_jbod</p
<p>pvname=3Dsdb</p
<p>ignore_vg_erros=3Dno<br
</p <p><br </p <p>[vg2]</p <p><b>action=3Dextend</b></p <p>vgname=3Dgluster_vg_jbod</p <p>pvname=3Dsdc</p
<p>ignore_vg_erros=3Dno<br
</p <br
<p>But will Gluster be able to auto-detect and use this SSD brick for tieri=
ng? Do I have to do some other configurations? Also, as the VM files (.qcow=
2) are quite big will I benefit from tiering? This is wrong and my approach=
should be other?</p <p><br </p <p>Thanks,</p <p>Moacir<br
</p <p><br </p <br <div
style=3D"color:rgb(49,55,57)" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Devin Acosta <de=
vin(a)pabstatencio.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 7:46 AM<br <b>To:</b> Moacir
Ferreira; users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">Moacir,</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">I have recently installed multiple Red Hat Virtualizatio=
n hosts for several different companies, and have dealt with the Red Hat Su=
pport Team in depth about optimal configuration
in regards to setting up GlusterFS most efficiently and I wanted to share =
with you what I learned.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">In general Red Hat Virtualization team frowns upon using=
each DISK of the system as just a JBOD, sure there is some protection by h=
aving the data replicated, however, the
recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 =
at the very least.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><br </div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<br <div
id=3D"bloop_sign_1502087376725469184" class=3D"bloop_sign"><span
style=
=3D"font-family:'helvetica Neue',helvetica;
font-size:14px">--</span><br st=
yle=3D"font-family:'helvetica Neue',helvetica; font-size:14px" <div class=3D"gmail_signature"
style=3D"font-family:'helvetica Neue',helvet=
ica; font-size:14px" <div dir=3D"ltr" <div><br
</div <div>Devin Acosta, RHCA,
RHVCA</div <div>Red Hat Certified
Architect</div <div></div </div
</div </div <br <p
class=3D"airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (<=
a
href=3D"mailto:moacirferreira@hotmail.com">moacirferreira@hotmail.com</a>=
) wrote:</p <blockquote
type=3D"cite" class=3D"clean_bq"><span <div dir=3D"ltr"
<div></div <div <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p><span>I am willing to assemble a oVirt "pod", made
of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br </p
<p>This said, my first question is: How should I deploy GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <p>Moacir<br
</p </div _______________________________________________ <br Users mailing list <br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a> <br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
t.org/mailman/listinfo/users</a <br
</div </div </span></blockquote
</div </div </div
_______________________________________________<br Users
mailing list<br <a class=3D"aqm-autolink
aqm-autowrap" href=3D"mailto:Users%40ovirt.org">Us=
ers(a)ovirt.org</a><br <a class=3D"aqm-autolink
aqm-autowrap" href=3D"http://lists.ovirt.org/mailm=
an/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users&... <br </blockquote </div
</div </div </div
</div </body </html
--_000_DB6P190MB0280BD0BB439D0C8A4588D00C88A0DB6P190MB0280EURP_--
Here is the direct quote from Red Hat when I asked about RAID
and Bri=
cks:</font></div<i><br>
</i></font></div<i>"A typical Gluster configuration would use RAID
underneath the bri=
cks. RAID 6 is most typical as it gives you 2 disk failure
protection, but =
RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired replication on top of that. Th=
e most popular way of doing this would be distributed replicated with 2x re=
plication. In general you'll get better performance with larger bricks=
. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all SSD=92s.=94 </i><=
/font></div<i>In order to SSD tiering from my understanding you
would need 1 x N=
VMe drive in each server, or 4 x SSD hot tier (it needs to be
distributed, =
replicated for the hot tier if not using NVME).
So with you only having 1 SSD drive in each server, I=92d suggest may=
be looking into the NVME option. </i></font></div<i><br>
</i></font></div<i>Since your using only 3-servers, what I=92d probably suggest
is to do (=
2 Replicas + Arbiter Node), this setup actually doesn=92t
require the 3=
rd server to have big drives at all as it only
stores meta-data about the files and not actually a full copy. </i></=
font></div<i><br>
</i></font></div<i>Please see the attached document that was given to me by Red
Hat to get=
more information on this. Hope this information helps
you.</i></font></div=
<i><br>
</i></font></div
1:24 p.m.
This is a multi-part message in MIME format.
------------15dc15fdd7045fa2768c750b77
Content-Type: text/plain; format=flowed; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On ovirt gluster uses sharding. So all large files are broken up in small=
=20
pieces on the gluster bricks.
/Johan
On August 8, 2017 12:19:39 Moacir Ferreira <moacirferreira(a)hotmail.com> w=
rote:
Thanks Johan, you brought "light" into my darkness! I went
looking for =
the=20
GlusterFS tiering how-to and it looks like quite simple to attach a
SSD=
as=20
hot tier. For those willing to read about it, go here:=20
http://blog.gluster.org/2016/03/automated-tiering-in-gluster/
Now, I still have a question: VMs are made of very large .qcow2 files. =
My=20
understanding is that files in Gluster are kept all together in a
singl=
e=20
brick. If so, I will not benefit from tiering as a single SSD will
not =
be=20
big enough to fit all my large VM .qcow2 files. This would not be
true =
if=20
Gluster can store "blocks" of data that compose a large
file spread on=20
several bricks. But if I am not wrong, this is one of key differences i=
n=20
between GlusterFS and Ceph. Can you comment?
Moacir
________________________________
From: Johan Bernhardsson <johan(a)kafit.se>
Sent: Tuesday, August 8, 2017 7:03 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
You attach the ssd as a hot tier with a gluster command. I don't think =
that=20
gdeploy or ovirt gui can do it.
The gluster docs and redhat docs explains tiering quite good.
/Johan
On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com>=
wrote:
Hi Devin,
Please consider that for the OS I have a RAID 1. Now, lets say I use RA=
ID 5=20
to assemble a single disk on each server. In this case, the SSD will
no=
t=20
make any difference, right? I guess that to be possible to use it,
the =
SSD=20
should not be part of the RAID 5. In this case I could create a
logical=
=20
volume made of the RAIDed brick and then extend it using the SSD.
I.e.:=
=20
Using gdeploy:
[disktype]
jbod
....
[pv1]
action=3Dcreate
devices=3Dsdb, sdc
wipefs=3Dyes
ignore_vg_erros=3Dno
[vg1]
action=3Dcreate
vgname=3Dgluster_vg_jbod
pvname=3Dsdb
ignore_vg_erros=3Dno
[vg2]
action=3Dextend
vgname=3Dgluster_vg_jbod
pvname=3Dsdc
ignore_vg_erros=3Dno
But will Gluster be able to auto-detect and use this SSD brick for tier=
ing?=20
Do I have to do some other configurations? Also, as the VM files
(.qcow=
2)=20
are quite big will I benefit from tiering? This is wrong and my
approac=
h=20
should be other?
Thanks,
Moacir
________________________________
From: Devin Acosta <devin(a)pabstatencio.com>
Sent: Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for sev=
eral=20
different companies, and have dealt with the Red Hat Support Team in
de=
pth=20
about optimal configuration in regards to setting up GlusterFS
most=20
efficiently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of t=
he=20
system as just a JBOD, sure there is some protection by having the
data=
=20
replicated, however, the recommendation is to use RAID 6 (preferred)
or=
=20
RAID-5, or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Brick=
s:
"A typical Gluster configuration would use RAID underneath the bricks. =
RAID=20
6 is most typical as it gives you 2 disk failure protection, but RAID
5=
=20
could be used too. Once you have the RAIDed bricks, you'd then
apply th=
e=20
desired replication on top of that. The most popular way of doing
this=20
would be distributed replicated with 2x replication. In general you'll =
get=20
better performance with larger bricks. 12 drives is often a sweet
spot.=
=20
Another option would be to create a separate tier using all
SSD=E2=80=99=
s.=E2=80=9D
In order to SSD tiering from my understanding you would need 1 x NVMe d=
rive=20
in each server, or 4 x SSD hot tier (it needs to be distributed,
replic=
ated=20
for the hot tier if not using NVME). So with you only having 1 SSD
driv=
e in=20
each server, I=E2=80=99d suggest maybe looking into the NVME option.
Since your using only 3-servers, what I=E2=80=99d probably suggest is t=
o do
(2=20
Replicas + Arbiter Node), this setup actually doesn=E2=80=99t require
t=
he 3rd=20
server to have big drives at all as it only stores meta-data about
the=20
files and not actually a full copy.
Please see the attached document that was given to me by Red Hat to get=
=20
more information on this. Hope this information helps you.
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira=20
(moacirferreira@hotmail.com<mailto:moacirferreira@hotmail.com>) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 =
CPU=20
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to
use=20
GlusterFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC=
and=20
a dual 10Gb NIC. So my intention is to create a loop like a server
tria=
ngle=20
using the 40Gb NICs for virtualization files (VMs .qcow2) access and
to=
=20
move VMs around the pod (east /west traffic) while using the 10Gb=20
interfaces for giving services to the outside world (north/south traffi=
c).
This said, my first question is: How should I deploy GlusterFS in such=20
oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and =
then=20
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not=20
consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the=20
virtualization environment, benefits from tiering when using a SSD disk=
?=20
And yes, will Gluster do it by default or I have to configure it to
do =
so?
At the bottom line, what is the good practice for using GlusterFS in sm=
all=20
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users%40ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
------------15dc15fdd7045fa2768c750b77
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<style type=3D"text/css" style=3D"display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
</head>
<body>
<div style=3D"color: black;">
<div style=3D"color: black;">
<p style=3D"margin: 0 0 1em 0; color: black;">On ovirt gluster uses
shard=
ing.
So all large files are broken up in small pieces on the gluster bricks.</=
p>
<p style=3D"margin: 0 0 1em 0; color: black;">/Johan</p>
</div>
<div style=3D"color: black;">
<p
style=3D"color: black; font-size: 10pt; font-family: Arial, sans-serif; m=
argin: 10pt 0;">On
August 8, 2017 12:19:39 Moacir Ferreira &lt;moacirferreira(a)hotmail.com&gt=
;
wrote:</p>
<blockquote type=3D"cite" class=3D"gmail_quote"
style=3D"margin: 0 0 0 0.75ex; border-left: 1px solid #808080; padding-le=
ft: 0.75ex;">
<div id=3D"divtagdefaultwrapper"
style=3D"font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-=
serif;"
dir=3D"ltr">
<p>Thanks Johan, you brought "light" into my darkness! I went
looking for the GlusterFS tiering how-to and it looks like quite simple t=
o
attach a SSD as hot tier. For those willing to read about it, go here:
<a href=3D"http://blog.gluster.org/2016/03/automated-tiering-in-gluster...
class=3D"OWAAutoLink" id=3D"LPlnk939343"
previewremoved=3D"true">
http://blog.gluster.org/2016/03/automated-tiering-in-gluster/</a>&l...
<p><br>
</p>
<p>Now, I still have a question: VMs are made of very large .qcow2 files.
My understanding is that files in Gluster are kept all together in a sing=
le
brick. If so, I will not benefit from tiering as a single SSD will not be
big enough to fit all my large VM
.qcow2 files. This would not be true if Gluster can store
"blocks" of data that compose a large file spread on several
bricks. But if I am not wrong, this is one of key differences in between
GlusterFS and Ceph. Can you comment?</p>
<p><br>
</p>
<p>Moacir<br>
</p>
<br>
<br>
<div style=3D"color: rgb(49, 55, 57);">
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt"
face=3D"Calibri, sans-serif"
color=3D"#000000"><b>From:</b> Johan Bernhar=
dsson
&lt;johan(a)kafit.se&gt;<br>
<b>Sent:</b> Tuesday, August 8, 2017 7:03 AM<br>
<b>To:</b> Moacir Ferreira; Devin Acosta; users(a)ovirt.org<br>
<b>Subject:</b> Re: [ovirt-users] Good practices</font>
<div> </div>
</div>
<div>
<div style=3D"color:black">
<div style=3D"color:black">
<p style=3D"margin:0 0 1em 0; color:black">You attach the ssd as a hot
ti=
er
with a gluster command. I don't think that gdeploy or ovirt gui can do it=
.</p>
<p style=3D"margin:0 0 1em 0; color:black">The gluster docs and redhat
do=
cs
explains tiering quite good.</p>
<p style=3D"margin:0 0 1em 0; color:black">/Johan</p>
</div>
<div style=3D"color:black">
<p
style=3D"color:black; font-size:10pt; font-family:Arial,sans-serif; margi=
n:10pt 0">
On August 8, 2017 07:06:42 Moacir Ferreira
&lt;moacirferreira(a)hotmail.com&gt; wrote:</p>
<blockquote type=3D"cite" class=3D"gmail_quote"
style=3D"margin:0 0 0 0.75ex; border-left:1px solid #808080; padding-left=
:0.75ex">
<div id=3D"divtagdefaultwrapper" dir=3D"ltr"
style=3D"font-size:12pt; color:#000000; font-family:Calibri,Helvetica,san=
s-serif">
<p>Hi Devin,</p>
<p><br>
</p>
<p>Please consider that for the OS I have a RAID 1. Now, lets say I use
RAID 5 to assemble a single disk on each server. In this case, the SSD wi=
ll
not make any difference, right? I guess that to be possible to use it, th=
e
SSD should not be part of the RAID
5. In this case I could create a logical volume made of the RAIDed brick
and then extend it using the SSD. I.e.: Using gdeploy:</p>
<p><br>
</p>
<p>[disktype]</p>
<p>jbod</p>
<p>....</p>
<p>[pv1]</p>
<p>action=3Dcreate</p>
<p>devices=3Dsdb, sdc</p>
<p>wipefs=3Dyes</p>
<p></p>
<p>ignore_vg_erros=3Dno<br>
</p>
<p><br>
</p>
<p></p>
<p>[vg1]</p>
<p><b>action=3Dcreate</b></p>
<p>vgname=3Dgluster_vg_jbod</p>
<p>pvname=3Dsdb</p>
<p>ignore_vg_erros=3Dno<br>
</p>
<p><br>
</p>
<p>[vg2]</p>
<p><b>action=3Dextend</b></p>
<p>vgname=3Dgluster_vg_jbod</p>
<p>pvname=3Dsdc</p>
<p>ignore_vg_erros=3Dno<br>
</p>
<br>
<p>But will Gluster be able to auto-detect and use this SSD brick for
tiering? Do I have to do some other configurations? Also, as the VM files
(.qcow2) are quite big will I benefit from tiering? This is wrong and my
approach should be other?</p>
<p><br>
</p>
<p>Thanks,</p>
<p>Moacir<br>
</p>
<p><br>
</p>
<br>
<div style=3D"color:rgb(49,55,57)">
<hr tabindex=3D"-1" style=3D"display:inline-block; width:98%">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt"
face=3D"Calibri, sans-serif"
color=3D"#000000"><b>From:</b> Devin Acosta
&lt;devin(a)pabstatencio.com&gt;<br>
<b>Sent:</b> Monday, August 7, 2017 7:46 AM<br>
<b>To:</b> Moacir Ferreira; users(a)ovirt.org<br>
<b>Subject:</b> Re: [ovirt-users] Good practices</font>
<div> </div>
</div>
<div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono">Moacir,</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono">I have recently installed multiple Red Hat Virtualiza=
tion
hosts for several different companies, and have dealt with the Red Hat
Support Team in depth about optimal configuration
in regards to setting up GlusterFS most efficiently and I wanted to shar=
e
with you what I learned.</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono">In general Red Hat Virtualization team frowns upon us=
ing
each DISK of the system as just a JBOD, sure there is some protection by
having the data replicated, however, the
recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-=
1
at the very least.</font></div>
<div id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font
face=3D"Input Mono"><br>
</font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o">Here
is the direct quote from Red Hat when I asked about RAID and
Bricks:</font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"><i>"A typical Gluster configuration would
use RA=
ID
underneath the bricks. RAID 6 is most typical as it gives you 2 disk
failure protection, but RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired replication on top of that.
The most popular way of doing this would be distributed replicated with 2=
x
replication. In general you'll get better performance with larger
bricks. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all
SSD=E2=80=99s.=E2=80=9D </i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><br>
</div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i>In
order to SSD tiering from my understanding you would need 1 x NVMe
drive in each server, or 4 x SSD hot tier (it needs to be distributed,
replicated for the hot tier if not using NVME).
So with you only having 1 SSD drive in each server, I=E2=80=99d sug=
gest maybe
looking into the NVME option. </i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"><i>Since your using only 3-servers, what
I=E2=80=99d =
probably
suggest is to do (2 Replicas + Arbiter Node), this setup actually
doesn=E2=80=99t require the 3rd server to have big drives at all as it on=
ly
stores meta-data about the files and not actually a full
copy. </i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"><i>Please see the attached document that was given
to=
me
by Red Hat to get more information on this. Hope this information helps
you.</i></font></div>
<div id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mon=
o"><i><br>
</i></font></div>
<br>
<div id=3D"bloop_sign_1502087376725469184"
class=3D"bloop_sign"><span
style=3D"font-family:'helvetica Neue',helvetica;
font-size:14px">--</span=
<br
style=3D"font-family:'helvetica
Neue',helvetica; font-size:14px">
<div class=3D"gmail_signature"
style=3D"font-family:'helvetica Neue',helvetica; font-size:14px">
<div dir=3D"ltr">
<div><br>
</div>
<div>Devin Acosta, RHCA, RHVCA</div>
<div>Red Hat Certified Architect</div>
<div></div>
</div>
</div>
</div>
<br>
<p class=3D"airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir Ferreira
=
(<a
href=3D"mailto:moacirferreira@hotmail.com">moacirferreira@hotmail.com</a>=
)
wrote:</p>
<blockquote type=3D"cite" class=3D"clean_bq"><span>
<div dir=3D"ltr">
<div></div>
<div>
<div id=3D"divtagdefaultwrapper" dir=3D"ltr"
style=3D"font-size:12pt; color:#000000; font-family:Calibri,Helvetica,san=
s-serif">
<p><span>I am willing to assemble a oVirt "pod", made of 3
servers, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD=
.
The idea is to use GlusterFS to provide HA for the VMs. The 3 servers hav=
e
a dual 40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for
virtualization files (VMs .qcow2) access and to move VMs around the pod
(east /west traffic) while using the 10Gb interfaces for giving services =
to
the outside world (north/south traffic).</span></p>
<p><br>
</p>
<p>This said, my first question is: How should I deploy GlusterFS in such
oVirt scenario? My questions are:</p>
<p><br>
</p>
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and
then create a GlusterFS using them?</p>
<p>2 - Instead, should I create a JBOD array made of all server's disks?<=
/p>
<p>3 - What is the best Gluster configuration to provide for HA while not
consuming too much disk space?<br>
</p>
<p>4 - Does a oVirt hypervisor pod like I am planning to build, and the
virtualization environment, benefits from tiering when using a SSD disk?
And yes, will Gluster do it by default or I have to configure it to do so=
?</p>
<p><br>
</p>
<p>At the bottom line, what is the good practice for using GlusterFS in
small pods for enterprises?<br>
</p>
<p><br>
</p>
<p>You opinion/feedback will be really appreciated!</p>
<p>Moacir<br>
</p>
</div>
_______________________________________________ <br>
Users mailing list <br>
<a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a> <br>
<a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.org/mailman/listinfo/users</a>
<br>
</div>
</div>
</span></blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Users mailing list<br>
<a class=3D"aqm-autolink aqm-autowrap"
href=3D"mailto:Users%40ovirt.org">Users@ovirt.org</a><br>
<a class=3D"aqm-autolink aqm-autowrap"
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.org/mailman/listinfo/users</a><br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>
------------15dc15fdd7045fa2768c750b77--
2:39 p.m.
--_000_DB6P190MB0280F066762EF81DE4270280C88A0DB6P190MB0280EURP_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Thanks once again Johan!
What would be your approach: JBOD straight or JBOD made of RAIDed bricks?
Moacir
________________________________
From: Johan Bernhardsson <johan(a)kafit.se Sent:
Tuesday, August 8, 2017 11:24 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
On ovirt gluster uses sharding. So all large files are broken up in small p=
ieces on the gluster bricks.
/Johan
On August 8, 2017 12:19:39 Moacir Ferreira <moacirferreira(a)hotmail.com> wro=
te:
Thanks Johan, you brought "light" into my darkness! I went looking for the =
GlusterFS tiering how-to and it looks like quite simple to attach a SSD as =
hot tier. For those willing to read about it, go here: http://blog.gluster.=
org/2016/03/automated-tiering-in-gluster/
Now, I still have a question: VMs are made of very large .qcow2 files. My u=
nderstanding is that files in Gluster are kept all together in a single bri=
ck. If so, I will not benefit from tiering as a single SSD will not be big =
enough to fit all my large VM .qcow2 files. This would not be true if Glust=
er can store "blocks" of data that compose a large file spread on several b=
ricks. But if I am not wrong, this is one of key differences in between Glu=
sterFS and Ceph. Can you comment?
Moacir
________________________________
From: Johan Bernhardsson <johan(a)kafit.se Sent:
Tuesday, August 8, 2017 7:03 AM
To: Moacir Ferreira; Devin Acosta; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
You attach the ssd as a hot tier with a gluster command. I don't think that=
gdeploy or ovirt gui can do it.
The gluster docs and redhat docs explains tiering quite good.
/Johan
On August 8, 2017 07:06:42 Moacir Ferreira <moacirferreira(a)hotmail.com> wro=
te:
Hi Devin,
Please consider that for the OS I have a RAID 1. Now, lets say I use RAID 5=
to assemble a single disk on each server. In this case, the SSD will not m=
ake any difference, right? I guess that to be possible to use it, the SSD s=
hould not be part of the RAID 5. In this case I could create a logical volu=
me made of the RAIDed brick and then extend it using the SSD. I.e.: Using g=
deploy:
[disktype]
jbod
....
[pv1]
action=3Dcreate
devices=3Dsdb, sdc
wipefs=3Dyes
ignore_vg_erros=3Dno
[vg1]
action=3Dcreate
vgname=3Dgluster_vg_jbod
pvname=3Dsdb
ignore_vg_erros=3Dno
[vg2]
action=3Dextend
vgname=3Dgluster_vg_jbod
pvname=3Dsdc
ignore_vg_erros=3Dno
But will Gluster be able to auto-detect and use this SSD brick for tiering?=
Do I have to do some other configurations? Also, as the VM files (.qcow2) =
are quite big will I benefit from tiering? This is wrong and my approach sh=
ould be other?
Thanks,
Moacir
________________________________
From: Devin Acosta <devin(a)pabstatencio.com Sent:
Monday, August 7, 2017 7:46 AM
To: Moacir Ferreira; users(a)ovirt.org
Subject: Re: [ovirt-users] Good practices
Moacir,
I have recently installed multiple Red Hat Virtualization hosts for several=
different companies, and have dealt with the Red Hat Support Team in depth=
about optimal configuration in regards to setting up GlusterFS most effici=
ently and I wanted to share with you what I learned.
In general Red Hat Virtualization team frowns upon using each DISK of the s=
ystem as just a JBOD, sure there is some protection by having the data repl=
icated, however, the recommendation is to use RAID 6 (preferred) or RAID-5,=
or at least RAID-1 at the very least.
Here is the direct quote from Red Hat when I asked about RAID and Bricks:
"A typical Gluster configuration would use RAID underneath the bricks. RAID=
6 is most typical as it gives you 2 disk failure protection, but RAID 5 co=
uld be used too. Once you have the RAIDed bricks, you'd then apply the desi=
red replication on top of that. The most popular way of doing this would be=
distributed replicated with 2x replication. In general you'll get better p=
erformance with larger bricks. 12 drives is often a sweet spot. Another opt=
ion would be to create a separate tier using all SSD=92s.=94
In order to SSD tiering from my understanding you would need 1 x NVMe drive=
in each server, or 4 x SSD hot tier (it needs to be distributed, replicate=
d for the hot tier if not using NVME). So with you only having 1 SSD drive =
in each server, I=92d suggest maybe looking into the NVME option.
Since your using only 3-servers, what I=92d probably suggest is to do (2 Re=
plicas + Arbiter Node), this setup actually doesn=92t require the 3rd serve=
r to have big drives at all as it only stores meta-data about the files and=
not actually a full copy.
Please see the attached document that was given to me by Red Hat to get mor=
e information on this. Hope this information helps you.
--
Devin Acosta, RHCA, RHVCA
Red Hat Certified Architect
On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (moacirferreira(a)hotmail.co=
m<mailto:moacirferreira@hotmail.com>) wrote:
I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU =
sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste=
rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua=
l 10Gb NIC. So my intention is to create a loop like a server triangle usin=
g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM=
s around the pod (east /west traffic) while using the 10Gb interfaces for g=
iving services to the outside world (north/south traffic).
This said, my first question is: How should I deploy GlusterFS in such oVir=
t scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then=
create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all server's disks?
3 - What is the best Gluster configuration to provide for HA while not cons=
uming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua=
lization environment, benefits from tiering when using a SSD disk? And yes,=
will Gluster do it by default or I have to configure it to do so?
At the bottom line, what is the good practice for using GlusterFS in small =
pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users%40ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_DB6P190MB0280F066762EF81DE4270280C88A0DB6P190MB0280EURP_
Content-Type: text/html; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3DWindows-1=
252" <style
type=3D"text/css" style=3D"display:none;"><!-- P
{margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr" <div
id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Helvetica,sans-serif;" dir=3D"ltr" <p>Thanks once again Johan!<br </p <br
<p>What would be your approach: JBOD straight or JBOD made of RAIDed bricks=
?</p <p><br
</p <p>Moacir<br </p <br <div
style=3D"color: rgb(49, 55, 57);" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Johan Bernhardsson =
&lt;johan(a)kafit.se&gt;<br
<b>Sent:</b> Tuesday, August 8, 2017 11:24 AM<br <b>To:</b> Moacir Ferreira; Devin Acosta;
users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
style=3D"color:black" <div
style=3D"color:black" <p style=3D"margin:0 0 1em
0; color:black">On ovirt gluster uses sharding. =
So all large files are broken up in small pieces on the gluster bricks.</p <p style=3D"margin:0 0 1em 0;
color:black">/Johan</p </div <div style=3D"color:black" <p style=3D"color:black; font-size:10pt;
font-family:Arial,sans-serif; marg=
in:10pt 0" On August 8, 2017 12:19:39 Moacir
Ferreira &lt;moacirferreira(a)hotmail.com&g=
t; wrote:</p <blockquote
type=3D"cite" class=3D"gmail_quote" style=3D"margin:0 0 0 0.75e=
x; border-left:1px solid #808080; padding-left:0.75ex" <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p>Thanks Johan, you brought "light" into my darkness! I
went loo=
king for the GlusterFS tiering how-to and it looks like quite simple to att=
ach a SSD as hot tier. For those willing to read about it, go here:
<a href=3D"http://blog.gluster.org/2016/03/automated-tiering-in-gluster... c=
lass=3D"OWAAutoLink" id=3D"LPlnk939343"
previewremoved=3D"true"
http://blog.gluster.org/2016/03/automated-tiering-in-gluster/</a>&l... <p><br
</p <p>Now, I still have a
question: VMs are made of very large .qcow2 files. M=
y understanding is that files in Gluster are kept all together in a single =
brick. If so, I will not benefit from tiering as a single SSD will not be b=
ig enough to fit all my large VM
.qcow2 files. This would not be true if Gluster can store "blocks&quo=
t; of data that compose a large file spread on several bricks. But if I am =
not wrong, this is one of key differences in between GlusterFS and Ceph. Ca=
n you comment?</p <p><br </p <p>Moacir<br </p <br
<br <div
style=3D"color:rgb(49,55,57)" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Johan Bernhardsson =
&lt;johan(a)kafit.se&gt;<br
<b>Sent:</b> Tuesday, August 8, 2017 7:03 AM<br <b>To:</b> Moacir Ferreira; Devin Acosta;
users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
style=3D"color:black" <div
style=3D"color:black" <p style=3D"margin:0 0 1em
0; color:black">You attach the ssd as a hot tier=
with a gluster command. I don't think that gdeploy or ovirt gui can do it.=
</p <p style=3D"margin:0 0 1em 0; color:black">The
gluster docs and redhat docs=
explains tiering quite good.</p <p
style=3D"margin:0 0 1em 0; color:black">/Johan</p </div <div
style=3D"color:black" <p style=3D"color:black;
font-size:10pt; font-family:Arial,sans-serif; marg=
in:10pt 0" On August 8, 2017 07:06:42 Moacir
Ferreira &lt;moacirferreira(a)hotmail.com&g=
t; wrote:</p <blockquote
type=3D"cite" class=3D"gmail_quote" style=3D"margin:0 0 0 0.75e=
x; border-left:1px solid #808080; padding-left:0.75ex" <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p>Hi Devin,</p <p><br </p <p>Please consider that for
the OS I have a RAID 1. Now, lets say I use RAI=
D 5 to assemble a single disk on each server. In this case, the SSD will no=
t make any difference, right? I guess that to be possible to use it, the SS=
D should not be part of the RAID
5. In this case I could create a logical volume made of the RAIDed brick a=
nd then extend it using the SSD. I.e.: Using gdeploy:</p
<p><br </p
<p>[disktype]</p <p>jbod</p <p>....</p
<p>[pv1]</p <p>action=3Dcreate</p <p>devices=3Dsdb, sdc</p
<p>wipefs=3Dyes</p <p></p <p>ignore_vg_erros=3Dno<br </p <p><br </p <p></p <p>[vg1]</p
<p><b>action=3Dcreate</b></p
<p>vgname=3Dgluster_vg_jbod</p
<p>pvname=3Dsdb</p
<p>ignore_vg_erros=3Dno<br
</p <p><br </p <p>[vg2]</p <p><b>action=3Dextend</b></p <p>vgname=3Dgluster_vg_jbod</p <p>pvname=3Dsdc</p
<p>ignore_vg_erros=3Dno<br
</p <br
<p>But will Gluster be able to auto-detect and use this SSD brick for tieri=
ng? Do I have to do some other configurations? Also, as the VM files (.qcow=
2) are quite big will I benefit from tiering? This is wrong and my approach=
should be other?</p <p><br </p <p>Thanks,</p <p>Moacir<br
</p <p><br </p <br <div
style=3D"color:rgb(49,55,57)" <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
Devin Acosta <de=
vin(a)pabstatencio.com&gt;<br <b>Sent:</b> Monday,
August 7, 2017 7:46 AM<br <b>To:</b> Moacir
Ferreira; users(a)ovirt.org<br <b>Subject:</b> Re:
[ovirt-users] Good practices</font
<div> </div
</div <div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">Moacir,</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">I have recently installed multiple Red Hat Virtualizatio=
n hosts for several different companies, and have dealt with the Red Hat Su=
pport Team in depth about optimal configuration
in regards to setting up GlusterFS most efficiently and I wanted to share =
with you what I learned.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono">In general Red Hat Virtualization team frowns upon using=
each DISK of the system as just a JBOD, sure there is some protection by h=
aving the data replicated, however, the
recommendation is to use RAID 6 (preferred) or RAID-5, or at least RAID-1 =
at the very least.</font></div <div
id=3D"bloop_customfont" style=3D"color:rgb(0,0,0);
margin:0px"><font f=
ace=3D"Input Mono"><br
</font></div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><br </div <div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div
id=3D"bloop_customfont" style=3D"margin:0px"><font
face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<div id=3D"bloop_customfont"
style=3D"margin:0px"><font face=3D"Input Mono"=
<br <div
id=3D"bloop_sign_1502087376725469184" class=3D"bloop_sign"><span
style=
=3D"font-family:'helvetica Neue',helvetica;
font-size:14px">--</span><br st=
yle=3D"font-family:'helvetica Neue',helvetica; font-size:14px" <div class=3D"gmail_signature"
style=3D"font-family:'helvetica Neue',helvet=
ica; font-size:14px" <div dir=3D"ltr" <div><br
</div <div>Devin Acosta, RHCA,
RHVCA</div <div>Red Hat Certified
Architect</div <div></div </div
</div </div <br <p
class=3D"airmail_on">On August 6, 2017 at 7:29:29 PM, Moacir Ferreira (<=
a
href=3D"mailto:moacirferreira@hotmail.com">moacirferreira@hotmail.com</a>=
) wrote:</p <blockquote
type=3D"cite" class=3D"clean_bq"><span <div dir=3D"ltr"
<div></div <div <div
id=3D"divtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt;
color=
:#000000; font-family:Calibri,Helvetica,sans-serif"
<p><span>I am willing to assemble a oVirt "pod", made
of 3 server=
s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id=
ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual=
40Gb NIC and a dual 10Gb NIC. So my intention
is to create a loop like a server triangle using the 40Gb NICs for virtual=
ization files (VMs .qcow2) access and to move VMs around the pod (east /wes=
t traffic) while using the 10Gb interfaces for giving services to the outsi=
de world (north/south traffic).</span></p
<p><br </p
<p>This said, my first question is: How should I deploy GlusterFS in such o=
Virt scenario? My questions are:</p
<p><br </p
<p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t=
hen create a GlusterFS using them?</p
<p>2 - Instead, should I create a JBOD array made of all server's
disks?</p=
<p>3 - What is the best Gluster configuration to provide
for HA while not c=
onsuming too much disk space?<br
</p <p>4 - Does a oVirt hypervisor
pod like I am planning to build, and the vir=
tualization environment, benefits from tiering when using a SSD disk? And y=
es, will Gluster do it by default or I have to configure it to do so?</p <p><br
</p <p>At the bottom line, what is
the good practice for using GlusterFS in sma=
ll pods for enterprises?<br </p
<p><br </p
<p>You opinion/feedback will be really appreciated!</p <p>Moacir<br
</p </div _______________________________________________ <br Users mailing list <br <a
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a> <br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
t.org/mailman/listinfo/users</a <br
</div </div </span></blockquote
</div </div </div
_______________________________________________<br Users
mailing list<br <a class=3D"aqm-autolink
aqm-autowrap" href=3D"mailto:Users%40ovirt.org">Us=
ers(a)ovirt.org</a><br <a class=3D"aqm-autolink
aqm-autowrap" href=3D"http://lists.ovirt.org/mailm=
an/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users&... <br </blockquote </div
</div </div </div
</div </blockquote </div
</div </div </div
</div </body </html
--_000_DB6P190MB0280F066762EF81DE4270280C88A0DB6P190MB0280EURP_--
Here is the direct quote from Red Hat when I asked about RAID
and Bri=
cks:</font></div<i><br>
</i></font></div<i>"A typical Gluster configuration would use RAID
underneath the bri=
cks. RAID 6 is most typical as it gives you 2 disk failure
protection, but =
RAID 5 could be used too. Once you have the
RAIDed bricks, you'd then apply the desired replication on top of that. Th=
e most popular way of doing this would be distributed replicated with 2x re=
plication. In general you'll get better performance with larger bricks=
. 12 drives is often a sweet spot. Another
option would be to create a separate tier using all SSD=92s.=94 </i><=
/font></div<i>In order to SSD tiering from my understanding you
would need 1 x N=
VMe drive in each server, or 4 x SSD hot tier (it needs to be
distributed, =
replicated for the hot tier if not using NVME).
So with you only having 1 SSD drive in each server, I=92d suggest may=
be looking into the NVME option. </i></font></div<i><br>
</i></font></div<i>Since your using only 3-servers, what I=92d probably suggest
is to do (=
2 Replicas + Arbiter Node), this setup actually doesn=92t
require the 3=
rd server to have big drives at all as it only
stores meta-data about the files and not actually a full copy. </i></=
font></div<i><br>
</i></font></div<i>Please see the attached document that was given to me by Red
Hat to get=
more information on this. Hope this information helps
you.</i></font></div=
<i><br>
</i></font></div
2664
days inactive
2666
days old
37 comments
10 participants
participants (10)
-
Colin Coe -
Devin Acosta -
Erekle Magradze -
Fabrice Bacchella -
FERNANDO FREDIANI -
Johan Bernhardsson -
Karli Sjöberg -
Moacir Ferreira -
Pavel Gashev -
Yaniv Kaul