--_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Fernando, Let's see what people say... But this is what I understood Red Hat says is = the best performance model. This is the main reason to open this discussion= because as long as I can see, some of you in the community, do not agree. But when I think about a "distributed file system", that can make any numbe= r of copies you want, it does not make sense using a RAIDed brick, what it = makes sense is to use JBOD. Moacir ________________________________ From: fernando.frediani@upx.com.br <fernando.frediani@upx.com.br> on behalf= of FERNANDO FREDIANI <fernando.frediani@upx.com> Sent: Tuesday, August 8, 2017 3:08 AM To: Moacir Ferreira Cc: Colin Coe; users@ovirt.org Subject: Re: [ovirt-users] Good practices Moacir, I understand that if you do this type of configuration you will be = severely impacted on storage performance, specially for writes. Even if you= have a Hardware RAID Controller with Writeback cache you will have a signi= ficant performance penalty and may not fully use all the resources you ment= ioned you have. Fernando 2017-08-07 10:03 GMT-03:00 Moacir Ferreira <moacirferreira@hotmail.com<mail= to:moacirferreira@hotmail.com>>: Hi Colin, Take a look on Devin's response. Also, read the doc he shared that gives so= me hints on how to deploy Gluster. It is more like that if you want high-performance you should have the brick= s created as RAID (5 or 6) by the server's disk controller and them assembl= e a JBOD GlusterFS. The attached document is Gluster specific and not for o= Virt. But at this point I think that having SSD will not be a plus as using= the RAID controller Gluster will not be aware of the SSD. Regarding the OS= , my idea is to have a RAID 1, made of 2 low cost HDDs, to install it. So far, based on the information received I should create a single RAID 5 o= r 6 on each server and then use this disk as a brick to create my Gluster c= luster, made of 2 replicas + 1 arbiter. What is new for me is the detail th= at the arbiter does not need a lot of space as it only keeps meta data. Thanks for your response! Moacir ________________________________ From: Colin Coe <colin.coe@gmail.com<mailto:colin.coe@gmail.com>> Sent: Monday, August 7, 2017 12:41 PM To: Moacir Ferreira Cc: users@ovirt.org<mailto:users@ovirt.org> Subject: Re: [ovirt-users] Good practices Hi I just thought that you'd do hardware RAID if you had the controller or JBO= D if you didn't. In hindsight, a server with 40Gbps NICs is pretty likely = to have a hardware RAID controller. I've never done JBOD with hardware RAI= D. I think having a single gluster brick on hardware JBOD would be riskier= than multiple bricks, each on a single disk, but thats not based on anythi= ng other than my prejudices. I thought gluster tiering was for the most frequently accessed files, in wh= ich case all the VMs disks would end up in the hot tier. However, I have b= een wrong before... I just wanted to know where the OS was going as I didn't see it mentioned i= n the OP. Normally, I'd have the OS on a RAID1 but in your case thats a lo= t of wasted disk. Honestly, I think Yaniv's answer was far better than my own and made the im= portant point about having an arbiter. Thanks On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira <moacirferreira@hotmail.com= <mailto:moacirferreira@hotmail.com>> wrote: Hi Colin, I am in Portugal, so sorry for this late response. It is quite confusing fo= r me, please consider: 1 - What if the RAID is done by the server's disk controller, not by softwa= re? 2 - For JBOD I am just using gdeploy to deploy it. However, I am not using = the oVirt node GUI to do this. 3 - As the VM .qcow2 files are quite big, tiering would only help if made b= y an intelligent system that uses SSD for chunks of data not for the entire= .qcow2 file. But I guess this is a problem everybody else has. So, Do you = know how tiering works in Gluster? 4 - I am putting the OS on the first disk. However, would you do differentl= y? Moacir ________________________________ From: Colin Coe <colin.coe@gmail.com<mailto:colin.coe@gmail.com>> Sent: Monday, August 7, 2017 4:48 AM To: Moacir Ferreira Cc: users@ovirt.org<mailto:users@ovirt.org> Subject: Re: [ovirt-users] Good practices 1) RAID5 may be a performance hit- 2) I'd be inclined to do this as JBOD by creating a distributed disperse vo= lume on each server. Something like echo gluster volume create dispersevol disperse-data 5 redundancy 2 \ $(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "server${SERV= ER}:/brick/brick-${SERVER}${BRICK}/brick \c"; done; done) 3) I think the above. 4) Gluster does support tiering, but IIRC you'd need the same number of SSD= as spindle drives. There may be another way to use the SSD as a fast cach= e. Where are you putting the OS? Hope I understood the question... Thanks On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira <moacirferreira@hotmail.co= m<mailto:moacirferreira@hotmail.com>> wrote: I am willing to assemble a oVirt "pod", made of 3 servers, each with 2 CPU = sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The idea is to use Gluste= rFS to provide HA for the VMs. The 3 servers have a dual 40Gb NIC and a dua= l 10Gb NIC. So my intention is to create a loop like a server triangle usin= g the 40Gb NICs for virtualization files (VMs .qcow2) access and to move VM= s around the pod (east /west traffic) while using the 10Gb interfaces for g= iving services to the outside world (north/south traffic). This said, my first question is: How should I deploy GlusterFS in such oVir= t scenario? My questions are: 1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and then= create a GlusterFS using them? 2 - Instead, should I create a JBOD array made of all server's disks? 3 - What is the best Gluster configuration to provide for HA while not cons= uming too much disk space? 4 - Does a oVirt hypervisor pod like I am planning to build, and the virtua= lization environment, benefits from tiering when using a SSD disk? And yes,= will Gluster do it by default or I have to configure it to do so? At the bottom line, what is the good practice for using GlusterFS in small = pods for enterprises? You opinion/feedback will be really appreciated! Moacir _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users --_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> </head> <body dir=3D"ltr"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Helvetica,sans-serif;" dir=3D"ltr"> <p>Fernando,</p> <p><br> </p> <p>Let's see what people say... But this is what I understood Red Hat says = is the best performance model. This is the main reason to open this discuss= ion because as long as I can see, some of you in the community, do not agre= e.<br> </p> <br> <p>But when I think about a "distributed file system", that can m= ake any number of copies you want, it does not make sense using a RAIDed br= ick, what it makes sense is to use JBOD.</p> <p><br> </p> <p>Moacir<br> </p> <br> <div style=3D"color: rgb(49, 55, 57);"> <hr tabindex=3D"-1" style=3D"display:inline-block; width:98%"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" face= =3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b> fernando.frediani@u= px.com.br <fernando.frediani@upx.com.br> on behalf of FERNANDO FREDIA= NI <fernando.frediani@upx.com><br> <b>Sent:</b> Tuesday, August 8, 2017 3:08 AM<br> <b>To:</b> Moacir Ferreira<br> <b>Cc:</b> Colin Coe; users@ovirt.org<br> <b>Subject:</b> Re: [ovirt-users] Good practices</font> <div> </div> </div> <div> <div dir=3D"ltr"> <div>Moacir, I understand that if you do this type of configuration you wil= l be severely impacted on storage performance, specially for writes. Even i= f you have a Hardware RAID Controller with Writeback cache you will have a = significant performance penalty and may not fully use all the resources you mentioned you have.<br> <br> </div> Fernando<br> </div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">2017-08-07 10:03 GMT-03:00 Moacir Ferreira <span= dir=3D"ltr"> <<a href=3D"mailto:moacirferreira@hotmail.com" target=3D"_blank">moacirf= erreira@hotmail.com</a>></span>:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1= px #ccc solid; padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_1874859859842763104divtagdefaultwrapper" dir=3D"ltr" style=3D"= font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif"> <p>Hi Colin,</p> <p><br> </p> <p>Take a look on Devin's response. Also, read the doc he shared that gives= some hints on how to deploy Gluster.</p> <p><br> </p> <p>It is more like that if you want high-performance you should have the br= icks created as RAID (5 or 6) by the server's disk controller and them= assemble a JBOD GlusterFS. The attached document is Gluster specific and n= ot for oVirt. But at this point I think that having SSD will not be a plus as using the RAID controller Gluster wi= ll not be aware of the SSD. Regarding the OS, my idea is to have a RAID 1, = made of 2 low cost HDDs, to install it.</p> <p><br> </p> <p>So far, based on the information received I should create a si= ngle RAID 5 or 6 on each server and then use this disk as a brick to create= my Gluster cluster, made of 2 replicas + 1 arbiter. What is new for me= is the detail that the arbiter does not need a lot of space as it only keeps meta data.</p> <p><br> </p> <p>Thanks for your response!<br> </p> Moacir<br> <br> <div style=3D"color:rgb(49,55,57)"> <hr style=3D"display:inline-block; width:98%"> <div id=3D"m_1874859859842763104divRplyFwdMsg" dir=3D"ltr"><font style=3D"f= ont-size:11pt" face=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b> = Colin Coe <<a href=3D"mailto:colin.coe@gmail.com" target=3D"_blank">coli= n.coe@gmail.com</a>><br> <b>Sent:</b> Monday, August 7, 2017 12:41 PM <div> <div class=3D"h5"><br> <b>To:</b> Moacir Ferreira<br> <b>Cc:</b> <a href=3D"mailto:users@ovirt.org" target=3D"_blank">users@ovirt= .org</a><br> <b>Subject:</b> Re: [ovirt-users] Good practices</div> </div> </font> <div> </div> </div> <div> <div class=3D"h5"> <div> <div dir=3D"ltr">Hi <div><br> </div> <div>I just thought that you'd do hardware RAID if you had the controller o= r JBOD if you didn't. In hindsight, a server with 40Gbps NICs is pret= ty likely to have a hardware RAID controller. I've never done JBOD wi= th hardware RAID. I think having a single gluster brick on hardware JBOD would be riskier than multiple bricks, each= on a single disk, but thats not based on anything other than my prejudices= .</div> <div><br> </div> <div>I thought gluster tiering was for the most frequently accessed files, = in which case all the VMs disks would end up in the hot tier. However= , I have been wrong before...</div> <div><br> </div> <div>I just wanted to know where the OS was going as I didn't see it mentio= ned in the OP. Normally, I'd have the OS on a RAID1 but in your case = thats a lot of wasted disk.</div> <div><br> </div> <div>Honestly, I think Yaniv's answer was far better than my own and made t= he important point about having an arbiter. </div> <div><br> </div> <div>Thanks</div> </div> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Mon, Aug 7, 2017 at 5:56 PM, Moacir Ferreira = <span dir=3D"ltr"> <<a href=3D"mailto:moacirferreira@hotmail.com" target=3D"_blank">moacirf= erreira@hotmail.com</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1= px #ccc solid; padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_1874859859842763104m_4831886883305672718divtagdefaultwrapper" = dir=3D"ltr" style=3D"font-size:12pt; color:#000000; font-family:Calibri,Hel= vetica,sans-serif"> <p>Hi Colin,</p> <p><br> </p> <p><span>I am in Portugal</span>, so sorry for this late response. It is qu= ite confusing for me, please consider:</p> <p></p> <div><b><br> </b>1<b> - </b>What if the RAID is done by the server's disk controller, no= t by software?</div> <br> <p></p> <p>2 -<b> </b>For JBOD I am just using gdeploy to deploy it. However, I am = not using the oVirt node GUI to do this.</p> <p><br> </p> <p>3 -<b> </b>As the VM .qcow2 files are quite big, tiering would only= help if made by an intelligent system that uses SSD for chunks of data not= for the entire .qcow2 file. But I guess this is a problem everybody else h= as. So, Do you know how tiering works in Gluster?<br> </p> <p><br> </p> <p>4 - I am putting the OS on the first disk. However, would you do di= fferently?<br> </p> <p><br> </p> Moacir<br> <br> <div style=3D"color:rgb(49,55,57)"> <hr style=3D"display:inline-block; width:98%"> <div id=3D"m_1874859859842763104m_4831886883305672718divRplyFwdMsg" dir=3D"= ltr"><font style=3D"font-size:11pt" face=3D"Calibri, sans-serif" color=3D"#= 000000"><b>From:</b> Colin Coe <<a href=3D"mailto:colin.coe@gmail.com" t= arget=3D"_blank">colin.coe@gmail.com</a>><br> <b>Sent:</b> Monday, August 7, 2017 4:48 AM<br> <b>To:</b> Moacir Ferreira<br> <b>Cc:</b> <a href=3D"mailto:users@ovirt.org" target=3D"_blank">users@ovirt= .org</a><br> <b>Subject:</b> Re: [ovirt-users] Good practices</font> <div> </div> </div> <div> <div dir=3D"ltr">1) RAID5 may be a performance hit- <br> <div><br> </div> <div><span>2) I'd be inclined to do this as JBOD by creating a distributed = disperse volume on each server. Something like <div><br> </div> <div>echo gluster volume create dispersevol disperse-data 5 redundancy 2 \<= /div> <div>$(for SERVER in a b c; do for BRICK in $(seq 1 5); do echo -e "se= rver${SERVER}:/brick/brick-<wbr>${SERVER}${BRICK}/brick \c"; done; don= e)</div> <div><br> </div> </span> <div>3) I think the above. <b></b></div> <span> <div><br> </div> <div>4) Gluster does support tiering, but IIRC you'd need the same number o= f SSD as spindle drives. There may be another way to use the SSD as a= fast cache. </div> <div><br> </div> <div>Where are you putting the OS?</div> <div><br> </div> <div>Hope I understood the question...</div> <div><br> </div> <div>Thanks</div> </span></div> </div> <span> <div class=3D"gmail_extra"><br> <div class=3D"gmail_quote">On Sun, Aug 6, 2017 at 10:49 PM, Moacir Ferreira= <span dir=3D"ltr"> <<a href=3D"mailto:moacirferreira@hotmail.com" target=3D"_blank">moacirf= erreira@hotmail.com</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex; border-left:1= px #ccc solid; padding-left:1ex"> <div dir=3D"ltr"> <div id=3D"m_1874859859842763104m_4831886883305672718m_2460985691746498322d= ivtagdefaultwrapper" dir=3D"ltr" style=3D"font-size:12pt; color:#000000; fo= nt-family:Calibri,Helvetica,sans-serif"> <p><span>I am willing to assemble a oVirt "pod", made of 3 server= s, each with 2 CPU sockets of 12 cores, 256GB RAM, 7 HDD 10K, 1 SSD. The id= ea is to use GlusterFS to provide HA for the VMs. The 3 servers have a dual= 40Gb NIC and a dual 10Gb NIC. So my intention is to create a loop like a server triangle using the 40Gb NICs for virtual= ization files (VMs .qcow2) access and to move VMs around the pod (east /wes= t traffic) while using the 10Gb interfaces for giving services to the outsi= de world (north/south traffic).</span></p> <p><br> <span></span></p> <p>This said, my first question is: How should I deploy GlusterFS in such o= Virt scenario? My questions are:</p> <p><br> </p> <p>1 - Should I create 3 RAID (i.e.: RAID 5), one on each oVirt node, and t= hen create a GlusterFS using them?</p> <p>2 - Instead, should I create a JBOD array made of all server's disks?</p=
<p>3 - What is the best Gluster configuration to provide for HA while not c= onsuming too much disk space?<br> </p> <p>4 - Does a oVirt hypervisor pod like I am planning to build, and the vir= tualization environment, benefits from tiering when using a SSD disk? And y= es, will Gluster do it by default or I have to configure it to do so?</p> <p><br> </p> <p>At the bottom line, what is the good practice for using GlusterFS in sma= ll pods for enterprises?<br> </p> <p><br> </p> <p>You opinion/feedback will be really appreciated!</p> <span class=3D"m_1874859859842763104m_4831886883305672718HOEnZb"><font colo= r=3D"#888888"> <p>Moacir<br> </p> </font></span></div> </div> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org" target=3D"_blank">Users@ovirt.org</a><br=
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><= br> <br> </blockquote> </div> <br> </div> </span></div> </div> </div> </div> </blockquote> </div> <br> </div> </div> </div> </div> </div> </div> </div> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer= " target=3D"_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><= br> <br> </blockquote> </div> <br> </div> </div> </div> </div> </body> </html> --_000_DB6P190MB02801EA0892B38F503220896C88A0DB6P190MB0280EURP_--