4:31 p.m.
This is a multi-part message in MIME format.
--------------A95CF1C0E4312049A540BB6A
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Exactly Moacir, that is my point.
A proper Distributed FIlesystem should not rely on any type of RAID as
it can make its own redundancy without having to rely on any underneath
layer (look at CEPH). Using RAID may help with management and in certain
scenarios to replace a faulty disk, but at a cost, not cheap by the way.
That's why in terms of resourcing saving, if a replica 3 brings those
issues mentioned it is much worth to have a small arbiter somewhere
instead of wasting a significant amount of disk space.
Fernando
On 08/08/2017 06:09, Moacir Ferreira wrote:
Fernando,
Let's see what people say... But this is what I understood Red Hat
says is the best performance model. This is the main reason to open
this discussion because as long as I can see, some of you in the
community, do not agree.
But when I think about a "distributed file system", that can make any
number of copies you want, it does not make sense using a RAIDed
brick, what it makes sense is to use JBOD.
Moacir
------------------------------------------------------------------------
*From:* fernando.frediani(a)upx.com.br <fernando.frediani(a)upx.com.br> on
behalf of FERNANDO FREDIANI <fernando.frediani(a)upx.com>
*Sent:* Tuesday, August 8, 2017 3:08 AM
*To:* Moacir Ferreira
*Cc:* Colin Coe; users(a)ovirt.org
*Subject:* Re: [ovirt-users] Good practices
Moacir, I understand that if you do this type of configuration you
will be severely impacted on storage performance, specially for
writes. Even if you have a Hardware RAID Controller with Writeback
cache you will have a significant performance penalty and may not
fully use all the resources you mentioned you have.
Fernando
2017-08-07 10:03 GMT-03:00 Moacir Ferreira <moacirferreira(a)hotmail.com
<mailto:moacirferreira@hotmail.com>>:
Hi Colin,
Take a look on Devin's response. Also, read the doc he shared that
gives some hints on how to deploy Gluster.
It is more like that if you want high-performance you should have
the bricks created as RAID (5 or 6) by the server's disk
controller and them assemble a JBOD GlusterFS. The attached
document is Gluster specific and not for oVirt. But at this point
I think that having SSD will not be a plus as using the RAID
controller Gluster will not be aware of the SSD. Regarding the OS,
my idea is to have a RAID 1, made of 2 low cost HDDs, to install it.
So far, based on the information received I should create a single
RAID 5 or 6 on each server and then use this disk as a brick to
create my Gluster cluster, made of 2 replicas + 1 arbiter. What is
new for me is the detail that the arbiter does not need a lot of
space as it only keeps meta data.
Thanks for your response!
Moacir
------------------------------------------------------------------------
*From:* Colin Coe <colin.coe(a)gmail.com <mailto:colin.coe@gmail.com>>
*Sent:* Monday, August 7, 2017 12:41 PM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org <mailto:users@ovirt.org>
*Subject:* Re: [ovirt-users] Good practices
Hi
I just thought that you'd do hardware RAID if you had the
controller or JBOD if you didn't. In hindsight, a server with
40Gbps NICs is pretty likely to have a hardware RAID controller.
I've never done JBOD with hardware RAID. I think having a single
gluster brick on hardware JBOD would be riskier than multiple
bricks, each on a single disk, but thats not based on anything
other than my prejudices.
I thought gluster tiering was for the most frequently accessed
files, in which case all the VMs disks would end up in the hot
tier. However, I have been wrong before...
I just wanted to know where the OS was going as I didn't see it
mentioned in the OP. Normally, I'd have the OS on a RAID1 but in
your case thats a lot of wasted disk.
Honestly, I think Yaniv's answer was far better than my own and
made the important point about having an arbiter.
Thanks
On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira
<moacirferreira(a)hotmail.com <mailto:moacirferreira@hotmail.com>>
wrote:
Hi Colin,
I am in Portugal, so sorry for this late response. It is quite
confusing for me, please consider:
*
*1*- *What if the RAID is done by the server's disk
controller, not by software?
2 -**For JBOD I am just using gdeploy to deploy it. However, I
am not using the oVirt node GUI to do this.
3 -**As the VM .qcow2 files are quite big, tiering would only
help if made by an intelligent system that uses SSD for chunks
of data not for the entire .qcow2 file. But I guess this is a
problem everybody else has. So, Do you know how tiering works
in Gluster?
4 - I am putting the OS on the first disk. However, would you
do differently?
Moacir
------------------------------------------------------------------------
*From:* Colin Coe <colin.coe(a)gmail.com
<mailto:colin.coe@gmail.com>>
*Sent:* Monday, August 7, 2017 4:48 AM
*To:* Moacir Ferreira
*Cc:* users(a)ovirt.org <mailto:users@ovirt.org>
*Subject:* Re: [ovirt-users] Good practices
1) RAID5 may be a performance hit-
2) I'd be inclined to do this as JBOD by creating a
distributed disperse volume on each server. Something like
echo gluster volume create dispersevol disperse-data 5
redundancy 2 \
$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e
"server${SERVER}:/brick/brick-${SERVER}${BRICK}/brick \c";
done; done)
3) I think the above.
4) Gluster does support tiering, but IIRC you'd need the same
number of SSD as spindle drives. There may be another way to
use the SSD as a fast cache.
Where are you putting the OS?
Hope I understood the question...
Thanks
On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira
<moacirferreira(a)hotmail.com
<mailto:moacirferreira@hotmail.com>> wrote:
I am willing to assemble a oVirt "pod", made of 3 servers,
each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K,
1 SSD. The idea is to use GlusterFS to provide HA for the
VMs. The 3 servers have a dual 40Gb NIC and a dual 10Gb
NIC. So my intention is to create a loop like a server
triangle using the 40Gb NICs for virtualization files (VMs
.qcow2) access and to move VMs around the pod (east /west
traffic) while using the 10Gb interfaces for giving
services to the outside world (north/south traffic).
This said, my first question is: How should I deploy
GlusterFS in such oVirt scenario? My questions are:
1 - Should I create 3 RAID (i.e.: RAID 5), one on each
oVirt node, and then create a GlusterFS using them?
2 - Instead, should I create a JBOD array made of all
server's disks?
3 - What is the best Gluster configuration to provide for
HA while not consuming too much disk space?
4 - Does a oVirt hypervisor pod like I am planning to
build, and the virtualization environment, benefits from
tiering when using a SSD disk? And yes, will Gluster do it
by default or I have to configure it to do so?
At the bottom line, what is the good practice for using
GlusterFS in small pods for enterprises?
You opinion/feedback will be really appreciated!
Moacir
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
<http://lists.ovirt.org/mailman/listinfo/users>
--------------A95CF1C0E4312049A540BB6A
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Exactly Moacir, that is my point.</p>
<p><br>
</p>
<p>A proper Distributed FIlesystem should not rely on any type of
RAID as it can make its own redundancy without having to rely on
any underneath layer (look at CEPH). Using RAID may help with
management and in certain scenarios to replace a faulty disk, but
at a cost, not cheap by the way.<br>
That's why in terms of resourcing saving, if a replica 3 brings
those issues mentioned it is much worth to have a small arbiter
somewhere instead of wasting a significant amount of disk space.</p>
<p><br>
Fernando<br>
</p>
<br>
<div class="moz-cite-prefix">On 08/08/2017 06:09, Moacir Ferreira
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DB6P190MB02801EA0892B38F503220896C88A0@DB6P190MB0280.EURP190.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"><!-- P
{margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
dir="ltr">
<p>Fernando,</p>
<p><br>
</p>
<p>Let's see what people say... But this is what I understood
Red Hat says is the best performance model. This is the main
reason to open this discussion because as long as I can see,
some of you in the community, do not agree.<br>
</p>
<br>
<p>But when I think about a "distributed file system", that can
make any number of copies you want, it does not make sense
using a RAIDed brick, what it makes sense is to use JBOD.</p>
<p><br>
</p>
<p>Moacir<br>
</p>
<br>
<div style="color: rgb(49, 55, 57);">
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
<a class="moz-txt-link-abbreviated"
href="mailto:fernando.frediani@upx.com.br">fernando.frediani@upx.com.br</a>
<a class="moz-txt-link-rfc2396E"
href="mailto:fernando.frediani@upx.com.br"><fernando.frediani@upx.com.br></a>
on behalf of FERNANDO
FREDIANI <a class="moz-txt-link-rfc2396E"
href="mailto:fernando.frediani@upx.com"><fernando.frediani@upx.com></a><br>
<b>Sent:</b> Tuesday, August 8, 2017 3:08 AM<br>
<b>To:</b> Moacir Ferreira<br>
<b>Cc:</b> Colin Coe; <a
class="moz-txt-link-abbreviated"
href="mailto:users@ovirt.org">users@ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] Good practices</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Moacir, I understand that if you do this type of
configuration you will be severely impacted on storage
performance, specially for writes. Even if you have a
Hardware RAID Controller with Writeback cache you will
have a significant performance penalty and may not fully
use all the resources you mentioned you have.<br>
<br>
</div>
Fernando<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2017-08-07 10:03 GMT-03:00 Moacir
Ferreira <span dir="ltr">
<<a href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira@hotmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div id="m_1874859859842763104divtagdefaultwrapper"
dir="ltr" style="font-size:12pt; color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>Hi Colin,</p>
<p><br>
</p>
<p>Take a look on Devin's response. Also, read the
doc he shared that gives some hints on how to
deploy Gluster.</p>
<p><br>
</p>
<p>It is more like that if you want
high-performance you should have the bricks
created as RAID (5 or 6) by the server's disk
controller and them assemble a JBOD GlusterFS.
The attached document is Gluster specific and
not for oVirt. But at this point I think that
having SSD will not be a plus as using the RAID
controller Gluster will not be aware of the SSD.
Regarding the OS, my idea is to have a RAID 1,
made of 2 low cost HDDs, to install it.</p>
<p><br>
</p>
<p>So far, based on the information received I
should create a single RAID 5 or 6 on each
server and then use this disk as a brick to
create my Gluster cluster, made of 2 replicas +
1 arbiter. What is new for me is the detail that
the arbiter does not need a lot of space as it
only keeps meta data.</p>
<p><br>
</p>
<p>Thanks for your response!<br>
</p>
Moacir<br>
<br>
<div style="color:rgb(49,55,57)">
<hr style="display:inline-block; width:98%">
<div id="m_1874859859842763104divRplyFwdMsg"
dir="ltr"><font
style="font-size:11pt"
color="#000000" face="Calibri,
sans-serif"><b>From:</b>
Colin Coe <<a
href="mailto:colin.coe@gmail.com"
target="_blank"
moz-do-not-send="true">colin.coe(a)gmail.com</a>&gt;<br>
<b>Sent:</b> Monday, August 7, 2017 12:41 PM
<div>
<div class="h5"><br>
<b>To:</b> Moacir Ferreira<br>
<b>Cc:</b> <a
href="mailto:users@ovirt.org"
target="_blank"
moz-do-not-send="true">users(a)ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] Good
practices</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="h5">
<div>
<div dir="ltr">Hi
<div><br>
</div>
<div>I just thought that you'd do
hardware RAID if you had the
controller or JBOD if you didn't. In
hindsight, a server with 40Gbps NICs
is pretty likely to have a hardware
RAID controller. I've never done JBOD
with hardware RAID. I think having a
single gluster brick on hardware JBOD
would be riskier than multiple bricks,
each on a single disk, but thats not
based on anything other than my
prejudices.</div>
<div><br>
</div>
<div>I thought gluster tiering was for
the most frequently accessed files, in
which case all the VMs disks would end
up in the hot tier. However, I have
been wrong before...</div>
<div><br>
</div>
<div>I just wanted to know where the OS
was going as I didn't see it mentioned
in the OP. Normally, I'd have the OS
on a RAID1 but in your case thats a
lot of wasted disk.</div>
<div><br>
</div>
<div>Honestly, I think Yaniv's answer
was far better than my own and made
the important point about having an
arbiter. </div>
<div><br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 7,
2017 at 5:56 PM, Moacir Ferreira <span
dir="ltr">
<<a
href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;
border-left:1px #ccc solid;
padding-left:1ex">
<div dir="ltr">
<div
id="m_1874859859842763104m_4831886883305672718divtagdefaultwrapper"
dir="ltr" style="font-size:12pt;
color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p>Hi Colin,</p>
<p><br>
</p>
<p><span>I am in
Portugal</span>,
so sorry for this late
response. It is quite
confusing for me, please
consider:</p>
<div><b><br>
</b>1<b> - </b>What if the
RAID is done by the server's
disk controller, not by
software?</div>
<br>
<p>2 -<b> </b>For JBOD I am
just using gdeploy to deploy
it. However, I am not using
the oVirt node GUI to do this.</p>
<p><br>
</p>
<p>3 -<b> </b>As the VM
.qcow2 files are quite big,
tiering would only help if
made by an intelligent system
that uses SSD for chunks of
data not for the entire .qcow2
file. But I guess this is a
problem everybody else has.
So, Do you know how tiering
works in Gluster?<br>
</p>
<p><br>
</p>
<p>4 - I am putting the OS on
the first disk. However, would
you do differently?<br>
</p>
<p><br>
</p>
Moacir<br>
<br>
<div style="color:rgb(49,55,57)">
<hr
style="display:inline-block;
width:98%">
<div
id="m_1874859859842763104m_4831886883305672718divRplyFwdMsg"
dir="ltr"><font
style="font-size:11pt"
color="#000000"
face="Calibri,
sans-serif"><b>From:</b>
Colin Coe <<a
href="mailto:colin.coe@gmail.com"
target="_blank"
moz-do-not-send="true">colin.coe(a)gmail.com</a>&gt;<br>
<b>Sent:</b> Monday,
August 7, 2017 4:48 AM<br>
<b>To:</b> Moacir
Ferreira<br>
<b>Cc:</b> <a
href="mailto:users@ovirt.org"
target="_blank"
moz-do-not-send="true">users(a)ovirt.org</a><br>
<b>Subject:</b> Re:
[ovirt-users] Good
practices</font>
<div> </div>
</div>
<div>
<div dir="ltr">1) RAID5 may
be a performance hit- <br>
<div><br>
</div>
<div><span>2) I'd be
inclined to do this as
JBOD by creating a
distributed disperse
volume on each
server. Something
like
<div><br>
</div>
<div>echo gluster
volume create
dispersevol
disperse-data 5
redundancy 2 \</div>
<div>$(for SERVER in a
b c; do for BRICK in
$(seq 1 5); do echo
-e
"server${SERVER}:/brick/brick-<wbr>${SERVER}${BRICK}/brick
\c"; done; done)</div>
<div><br>
</div>
</span>
<div>3) I think the
above. </div>
<span>
<div><br>
</div>
<div>4) Gluster does
support tiering, but
IIRC you'd need the
same number of SSD
as spindle drives.
There may be another
way to use the SSD
as a fast cache. </div>
<div><br>
</div>
<div>Where are you
putting the OS?</div>
<div><br>
</div>
<div>Hope I understood
the question...</div>
<div><br>
</div>
<div>Thanks</div>
</span></div>
</div>
<span>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Sun, Aug 6, 2017 at
10:49 PM, Moacir
Ferreira <span
dir="ltr">
<<a
href="mailto:moacirferreira@hotmail.com"
target="_blank"
moz-do-not-send="true">moacirferreira(a)hotmail.com</a>&gt;</span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;
border-left:1px #ccc
solid;
padding-left:1ex">
<div dir="ltr">
<div
id="m_1874859859842763104m_4831886883305672718m_2460985691746498322divtagdefaultwrapper"
dir="ltr"
style="font-size:12pt;
color:#000000;
font-family:Calibri,Helvetica,sans-serif">
<p><span>I am
willing to
assemble a
oVirt "pod",
made of 3
servers, each
with 2 CPU
sockets of 12
cores, 256GB
RAM, 7 HDD
10K, 1 SSD.
The idea is to
use GlusterFS
to provide HA
for the VMs.
The 3 servers
have a dual
40Gb NIC and a
dual 10Gb NIC.
So my
intention is
to create a
loop like a
server
triangle using
the 40Gb NICs
for
virtualization
files (VMs
.qcow2) access
and to move
VMs around the
pod (east
/west traffic)
while using
the 10Gb
interfaces for
giving
services to
the outside
world
(north/south
traffic).</span></p>
<p><br>
<span></span></p>
<p>This said, my
first question
is: How should
I deploy
GlusterFS in
such oVirt
scenario? My
questions are:</p>
<p><br>
</p>
<p>1 - Should I
create 3 RAID
(i.e.: RAID
5), one on
each oVirt
node, and then
create a
GlusterFS
using them?</p>
<p>2 - Instead,
should I
create a JBOD
array made of
all server's
disks?</p>
<p>3 - What is
the best
Gluster
configuration
to provide for
HA while not
consuming too
much disk
space?<br>
</p>
<p>4 - Does a
oVirt
hypervisor pod
like I am
planning to
build, and the
virtualization
environment,
benefits from
tiering when
using a SSD
disk? And yes,
will Gluster
do it by
default or I
have to
configure it
to do so?</p>
<p><br>
</p>
<p>At the bottom
line, what is
the good
practice for
using
GlusterFS in
small pods for
enterprises?<br>
</p>
<p><br>
</p>
<p>You
opinion/feedback
will be really
appreciated!</p>
<span
class="m_1874859859842763104m_4831886883305672718HOEnZb"><font
color="#888888">
<p>Moacir<br>
</p>
</font></span></div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a
href="mailto:Users@ovirt.org"
target="_blank"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/mailman<wbr...
<br>
</blockquote>
</div>
<br>
</div>
</span></div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.ovirt.org/<wbr>mai...
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>
--------------A95CF1C0E4312049A540BB6A--