recommendations for best performance and reliability
Hi, Can someone please give me some pointers, what would be the best setup for performance and reliability? We have the following hardware setup: 3x Supermicro server with following features per server: 128GB RAM 4x 8TB SATA HDD 2x SSD drives (intel_ssdsc2ba400g4 - 400GB DC S3710) 2x 12 core CPU (Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz Quad port 10Gbe Inter NIC 2x 10GB Cisco switches (to isolate storage network from LAN) One of the servers will be in another office, with a 600Mb wireless link for Disaster Recovery. What is recommended for the best setup in terms of redundancy and speed? I am guessing GlusterFS with a Distributed Striped Replicated Volume across 3 of the servers. For added performance I want to use the SSD drives, perhaps with dm-cache? Should I combine the 4x HDD's using LVM on each host node? What about RAID 6? Virtual Machines will then reside on the oVirt Cluster and any one of the 3 host nodes can fail, or any single HDD can fail and all should still work, right/? -- Kind Regards Rudi Ahlers Website: http://www.rudiahlers.co.za
This is a multi-part message in MIME format. --------------714A7C973919A95EE2A8FDD0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Helli Rudi Nice specs. I wouldn't use GlusterFS for this setup with the third server in a different location. Just have this server as an Standalone and replicate the VMs there. You won't have real time replication, but much less hassle and probably to have constant failures, specially knowing you have a wireless link. For the SSDs I have been using bcache with success. Relatively simple to setup and pretty good performance. For your specs as you have 4 mechanical disks I would recommend you to have a RAID 5 between them (4 disks is my limit for RAID 5) and a RAID 0 made of SSDs for the bcache device. If the RAID 0 fails for any reason it will fall back directly to the mechanical disks and you can do maintenance on the Node doing live migration in order to replace the failed disks. However as you have you have 2 remaining server to create your cluster you may need to consider GlusterFS on the top of this RAID to have the replication and Highavaibility. Hope it helps. Fernando On 13/11/2017 08:03, Rudi Ahlers wrote:
Hi,
Can someone please give me some pointers, what would be the best setup for performance and reliability?
We have the following hardware setup:
3x Supermicro server with following features per server: 128GB RAM 4x 8TB SATA HDD 2x SSD drives (intel_ssdsc2ba400g4 - 400GB DC S3710) 2x 12 core CPU (Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz Quad port 10Gbe Inter NIC 2x 10GB Cisco switches (to isolate storage network from LAN)
One of the servers will be in another office, with a 600Mb wireless link for Disaster Recovery.
What is recommended for the best setup in terms of redundancy and speed?
I am guessing GlusterFS with a Distributed Striped Replicated Volume across 3 of the servers.
For added performance I want to use the SSD drives, perhaps with dm-cache?
Should I combine the 4x HDD's using LVM on each host node? What about RAID 6?
Virtual Machines will then reside on the oVirt Cluster and any one of the 3 host nodes can fail, or any single HDD can fail and all should still work, right/?
-- Kind Regards Rudi Ahlers Website: http://www.rudiahlers.co.za
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------714A7C973919A95EE2A8FDD0 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Helli Rudi</p> <p>Nice specs.</p> <p>I wouldn't use GlusterFS for this setup with the third server in a different location. Just have this server as an Standalone and replicate the VMs there. You won't have real time replication, but much less hassle and probably to have constant failures, specially knowing you have a wireless link.</p> <p>For the SSDs I have been using bcache with success. Relatively simple to setup and pretty good performance.</p> <p>For your specs as you have 4 mechanical disks I would recommend you to have a RAID 5 between them (4 disks is my limit for RAID 5) and a RAID 0 made of SSDs for the bcache device. If the RAID 0 fails for any reason it will fall back directly to the mechanical disks and you can do maintenance on the Node doing live migration in order to replace the failed disks.</p> <p>However as you have you have 2 remaining server to create your cluster you may need to consider GlusterFS on the top of this RAID to have the replication and Highavaibility.</p> <p>Hope it helps.</p> <p>Fernando<br> </p> <br> <div class="moz-cite-prefix">On 13/11/2017 08:03, Rudi Ahlers wrote:<br> </div> <blockquote type="cite" cite="mid:CAAj3DjmsZOkkzTZ_wYz-B7_xV9YoKgknyD0r=RJu5=sa0i7=wg@mail.gmail.com"> <div dir="ltr">Hi, <div><br> </div> <div>Can someone please give me some pointers, what would be the best setup for performance and reliability?</div> <div><br> </div> <div>We have the following hardware setup:</div> <div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px"><br> </span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">3x Supermicro server with following features per server:</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">128GB RAM</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">4x 8TB SATA HDD</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">2x SSD drives (intel_ssdsc2ba400g4 - 400GB DC S3710)</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">2x 12 core CPU (Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">Quad port 10Gbe Inter NIC</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">2x 10GB Cisco switches (to isolate storage network from LAN)</span></font></div> </div> <div><br> </div> <div> <div>One of the servers will be in another office, with a 600Mb wireless link for Disaster Recovery. </div> <div><br> </div> <div>What is recommended for the best setup in terms of redundancy and speed?</div> <div><br> </div> <div>I am guessing GlusterFS with a Distributed Striped Replicated Volume across 3 of the servers. </div> <div><br> </div> <div>For added performance I want to use the SSD drives, perhaps with dm-cache?</div> <div><br> </div> <div>Should I combine the 4x HDD's using LVM on each host node?</div> <div>What about RAID 6?</div> <div><br> </div> <div><br> </div> <div><br> </div> <div> <div>Virtual Machines will then reside on the oVirt Cluster and any one of the 3 host nodes can fail, or any single HDD can fail and all should still work, right/? </div> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div><br> </div> -- <br> <div class="gmail_signature">Kind Regards<br> Rudi Ahlers<br> Website: <a href="http://www.rudiahlers.co.za" target="_blank" moz-do-not-send="true">http://www.rudiahlers.co.za</a><br> </div> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------714A7C973919A95EE2A8FDD0--
This is a multi-part message in MIME format. --------------20F62C253AB9FA2BFC1BDEA7 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Just curious. Would Gluster with a JBOD configuration make sense in this scenario? Maybe with dm-cache on the SSDs? In general, does a Gluster JBOD configuration make sense as a backend for oVirt? Eduardo Mayoral Jimeno (emayoral@arsys.es) Administrador de sistemas. Departamento de Plataformas. Arsys internet. +34 941 620 145 ext. 5153 On 13/11/17 12:47, FERNANDO FREDIANI wrote:
Helli Rudi
Nice specs.
I wouldn't use GlusterFS for this setup with the third server in a different location. Just have this server as an Standalone and replicate the VMs there. You won't have real time replication, but much less hassle and probably to have constant failures, specially knowing you have a wireless link.
For the SSDs I have been using bcache with success. Relatively simple to setup and pretty good performance.
For your specs as you have 4 mechanical disks I would recommend you to have a RAID 5 between them (4 disks is my limit for RAID 5) and a RAID 0 made of SSDs for the bcache device. If the RAID 0 fails for any reason it will fall back directly to the mechanical disks and you can do maintenance on the Node doing live migration in order to replace the failed disks.
However as you have you have 2 remaining server to create your cluster you may need to consider GlusterFS on the top of this RAID to have the replication and Highavaibility.
Hope it helps.
Fernando
On 13/11/2017 08:03, Rudi Ahlers wrote:
Hi,
Can someone please give me some pointers, what would be the best setup for performance and reliability?
We have the following hardware setup:
3x Supermicro server with following features per server: 128GB RAM 4x 8TB SATA HDD 2x SSD drives (intel_ssdsc2ba400g4 - 400GB DC S3710) 2x 12 core CPU (Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz Quad port 10Gbe Inter NIC 2x 10GB Cisco switches (to isolate storage network from LAN)
One of the servers will be in another office, with a 600Mb wireless link for Disaster Recovery.
What is recommended for the best setup in terms of redundancy and speed?
I am guessing GlusterFS with a Distributed Striped Replicated Volume across 3 of the servers.
For added performance I want to use the SSD drives, perhaps with dm-cache?
Should I combine the 4x HDD's using LVM on each host node? What about RAID 6?
Virtual Machines will then reside on the oVirt Cluster and any one of the 3 host nodes can fail, or any single HDD can fail and all should still work, right/?
-- Kind Regards Rudi Ahlers Website: http://www.rudiahlers.co.za
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------20F62C253AB9FA2BFC1BDEA7 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Just curious. Would Gluster with a JBOD configuration make sense in this scenario? Maybe with dm-cache on the SSDs?</p> <p>In general, does a Gluster JBOD configuration make sense as a backend for oVirt?<br> </p> <pre class="moz-signature" cols="72">Eduardo Mayoral Jimeno (<a class="moz-txt-link-abbreviated" href="mailto:emayoral@arsys.es">emayoral@arsys.es</a>) Administrador de sistemas. Departamento de Plataformas. Arsys internet. +34 941 620 145 ext. 5153</pre> <div class="moz-cite-prefix">On 13/11/17 12:47, FERNANDO FREDIANI wrote:<br> </div> <blockquote type="cite" cite="mid:8940c496-6d8d-ddf7-7b57-3f828f9102d2@upx.com"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <p>Helli Rudi</p> <p>Nice specs.</p> <p>I wouldn't use GlusterFS for this setup with the third server in a different location. Just have this server as an Standalone and replicate the VMs there. You won't have real time replication, but much less hassle and probably to have constant failures, specially knowing you have a wireless link.</p> <p>For the SSDs I have been using bcache with success. Relatively simple to setup and pretty good performance.</p> <p>For your specs as you have 4 mechanical disks I would recommend you to have a RAID 5 between them (4 disks is my limit for RAID 5) and a RAID 0 made of SSDs for the bcache device. If the RAID 0 fails for any reason it will fall back directly to the mechanical disks and you can do maintenance on the Node doing live migration in order to replace the failed disks.</p> <p>However as you have you have 2 remaining server to create your cluster you may need to consider GlusterFS on the top of this RAID to have the replication and Highavaibility.</p> <p>Hope it helps.</p> <p>Fernando<br> </p> <br> <div class="moz-cite-prefix">On 13/11/2017 08:03, Rudi Ahlers wrote:<br> </div> <blockquote type="cite" cite="mid:CAAj3DjmsZOkkzTZ_wYz-B7_xV9YoKgknyD0r=RJu5=sa0i7=wg@mail.gmail.com"> <div dir="ltr">Hi, <div><br> </div> <div>Can someone please give me some pointers, what would be the best setup for performance and reliability?</div> <div><br> </div> <div>We have the following hardware setup:</div> <div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px"><br> </span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">3x Supermicro server with following features per server:</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">128GB RAM</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">4x 8TB SATA HDD</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">2x SSD drives (intel_ssdsc2ba400g4 - 400GB DC S3710)</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">2x 12 core CPU (Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">Quad port 10Gbe Inter NIC</span></font></div> <div><font color="#2c2c2c" face="Open Sans, Helvetica, Arial, sans-serif"><span style="font-size:14.6667px">2x 10GB Cisco switches (to isolate storage network from LAN)</span></font></div> </div> <div><br> </div> <div> <div>One of the servers will be in another office, with a 600Mb wireless link for Disaster Recovery. </div> <div><br> </div> <div>What is recommended for the best setup in terms of redundancy and speed?</div> <div><br> </div> <div>I am guessing GlusterFS with a Distributed Striped Replicated Volume across 3 of the servers. </div> <div><br> </div> <div>For added performance I want to use the SSD drives, perhaps with dm-cache?</div> <div><br> </div> <div>Should I combine the 4x HDD's using LVM on each host node?</div> <div>What about RAID 6?</div> <div><br> </div> <div><br> </div> <div><br> </div> <div> <div>Virtual Machines will then reside on the oVirt Cluster and any one of the 3 host nodes can fail, or any single HDD can fail and all should still work, right/? </div> </div> <div><br> </div> <div><br> </div> <div><br> </div> <div><br> </div> -- <br> <div class="gmail_signature">Kind Regards<br> Rudi Ahlers<br> Website: <a href="http://www.rudiahlers.co.za" target="_blank" moz-do-not-send="true">http://www.rudiahlers.co.za</a><br> </div> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org" moz-do-not-send="true">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users" moz-do-not-send="true">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------20F62C253AB9FA2BFC1BDEA7--
On Mon, Nov 13, 2017 at 3:33 PM, Rudi Ahlers <rudiahlers@gmail.com> wrote:
Hi,
Can someone please give me some pointers, what would be the best setup for performance and reliability?
We have the following hardware setup:
3x Supermicro server with following features per server: 128GB RAM 4x 8TB SATA HDD 2x SSD drives (intel_ssdsc2ba400g4 - 400GB DC S3710) 2x 12 core CPU (Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz Quad port 10Gbe Inter NIC 2x 10GB Cisco switches (to isolate storage network from LAN)
One of the servers will be in another office, with a 600Mb wireless link for Disaster Recovery.
All 3 servers need to be on the same network with low latency between servers. Gluster volume is used in replica 3 configuration in oVirt, and data will be written to all 3 servers synchronously. If you have one of the 3 servers in a remote location, your writes will be slow and the storage will seem to be unavailable to oVirt. For Disaster recovery, you can use gluster's geo-replication feature. This does need an additional server in the remote location. See https://www.ovirt.org/develop/release-management/features/gluster/gluster-dr...
What is recommended for the best setup in terms of redundancy and speed?
I am guessing GlusterFS with a Distributed Striped Replicated Volume across 3 of the servers.
Replica 3 gluster volume with sharding turned on for the gluster volume. If you use the deployment via Cockpit option, the volume is created with all recommended options. You can also manually set this by enabling the virt profile, like this # gluster volume set <volumename> group virt
For added performance I want to use the SSD drives, perhaps with dm-cache?
Should I combine the 4x HDD's using LVM on each host node? What about RAID 6?
Gluster can work with either RAID or JBOD ( with replica 3, JBOD can be used, as gluster has the redundancy built in on the other nodes). Users often choose RAID to aggregate capacity from multiple disks as single brick, and also as the brick rebuilt time is offloaded to RAID layer in case of any of the hard disk failures. The answer is it depends on your needs and the hardware you have.
Virtual Machines will then reside on the oVirt Cluster and any one of the 3 host nodes can fail, or any single HDD can fail and all should still work, right/?
That's correct. With a replica 3 approach, the oVirt environment is available even when one of the 3 hosts fail.
-- Kind Regards Rudi Ahlers Website: http://www.rudiahlers.co.za
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Eduardo Mayoral -
FERNANDO FREDIANI -
Rudi Ahlers -
Sahina Bose