Hi, We are currently building a three node hyper-converged cluster based on oVirt Node and Gluster. While discussing the different storage layout we couldn't get to a final decision. Currently our servers are equipped as follows: - servers 1 & 2: - Two 800GB disks for OS - 100GB RAID1 used as LVM PV for OS - Nine 7.68TB disks for Gluster - 60TB RAID 5 used as LVM PV for Gluster - server 3 - Two 800GB disks for OS & Gluster - 100GB RAID 1 used as LVM PV for OS - 700GB RAID 1 used as LVM PV for Gluster Unfortunately I couldn't find much information about mdadm on this topic. The hyper-convergence guides ([1], [2]) seem to assume that there is either a hardware RAID in place or JBOD is used. Is there some documentation available on what to consider when using mdadm? Or would it be more sensible to just use JBOD and then add redundancy on the LVM or Gluster level? If choosing to go with mdadm, what option should I choose in the bricks wizard screen (RAID 5 or JBOD)? [1]: https://ovirt.org/documentation/gluster-hyperconverged/chap-Deploying_Hyperc... [2]: https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infrast...
Hi all, Feel free to give your personal opinion whether software RAID makes any sense at all with oVirt Node or what your architecture would be given the mentioned hardware (3 servers, 6 800GB NVME disks, 18 7.68TB NVME disks). Thanks a lot for any feedback, Jonas
On Thu, Jan 20, 2022 at 8:52 AM <jonas@rabe.ch> wrote:
Hi all,
Feel free to give your personal opinion whether software RAID makes any sense at all with oVirt Node or what your architecture would be given the mentioned hardware (3 servers, 6 800GB NVME disks, 18 7.68TB NVME disks).
Thanks a lot for any feedback, Jonas
Hello, I somehow missed your original email. While I don't have a comparable setup (We usually opt for mixed SSD HDD setups with fast VMs running on SSD bached gluster volumes and large-storage VMs using HDD backed gluster volume), I can offer some insight. 1. Performance wise, using software RAID10 on the NVME drives will give you great performance, while reducing the chance of a double drop. 2. Avoid using RAID5/50/6/60 as the write amplification will eat up your read-intensive NVMEs life span in a couple of months. 3, Use a fast network back-end. A good high-end server with hardware RAID HDD can easily saturate a 10GbR Linux @peak load. 4. Use arbiter volumes, split the load between the nodes to reduce the write amplifications. (N1: brick, brick, arbiter. N2: brick, arbiter, brick. N3: arbiter, brick, brick). 5. In general I usually avoid using the wizard, manually creating the gluster volumes before I begin the deployment. - Gilboa
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZZ5VDBPHTEVGAT...
On Fri, Jan 21, 2022 at 10:45 AM Gilboa Davara <gilboad@gmail.com> wrote:
On Thu, Jan 20, 2022 at 8:52 AM <jonas@rabe.ch> wrote:
Hi all,
Feel free to give your personal opinion whether software RAID makes any sense at all with oVirt Node or what your architecture would be given the mentioned hardware (3 servers, 6 800GB NVME disks, 18 7.68TB NVME disks).
Thanks a lot for any feedback, Jonas
Hello,
I somehow missed your original email. While I don't have a comparable setup (We usually opt for mixed SSD HDD setups with fast VMs running on SSD bached gluster volumes and large-storage VMs using HDD backed gluster volume), I can offer some insight. 1. Performance wise, using software RAID10 on the NVME drives will give you great performance, while reducing the chance of a double drop. 2. Avoid using RAID5/50/6/60 as the write amplification will eat up your read-intensive NVMEs life span in a couple of months. 3, Use a fast network back-end. A good high-end server with hardware RAID HDD can easily saturate a 10GbR Linux @peak load. 4. Use arbiter volumes, split the load between the nodes to reduce the write amplifications. (N1: brick, brick, arbiter. N2: brick, arbiter, brick. N3: arbiter, brick, brick). 5. In general I usually avoid using the wizard, manually creating the gluster volumes before I begin the deployment.
- Gilboa
I missed the fact that your 3'rd host has considerably smaller storage. In this case, luckily I have a comparable setup @Home. I simply placed all the arbiter volumes on the 3'rd host. - Gilboa
Using the wizzard is utilizing the Gluster Andible roles.I would highly recommend using it, unless you know what you are doing (for example storage alignment when using Hardware raid). Keep in mind that the DHT xlator (the logic in distributed volumes) is shard aware, so your shards are spread between subvolumes and additional performance can be gained.So using replicated-distributed volumes have their benefits. If you decide to avoid the software raid, use only replica3 volumes as with SSDs/NVMEs usually the failures are not physical, but logical (maximum writes reached -> predictive failure -> total failure). Also, consider mounting via noatime/relatime and context="system_u:object_r:glusterd_brick_t:s0" for your gluster bricks. Best Regards,Strahil Nikolov On Fri, Jan 21, 2022 at 11:00, Gilboa Davara<gilboad@gmail.com> wrote: _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/U2ZEWLRF5D6FEN...
participants (3)
-
Gilboa Davara -
jonas＠rabe.ch -
Strahil Nikolov