Vincent,
I've been back and forth on SSDs vs HDDs and can't really get a clear answer. You are correct though, it would only equal 4TB usable in the end which is pretty crazy but that amount of 7200 RPM HDDs equals about the same cost as 3 2TB ssds would. I actually posted a question to this list not long ago asking how GlusterFS might perform with a small amount of disks such as one 2TB SSD per host and some glusterFS users commented stating that network would be the bottleneck long before the disks and a small amount of SSDs could bottleneck at RPC layer. Also, I believe at this time GlusterFS is not exactly developed to take full advantage of SSDs (but I believe there has been strides being made in that regard, I could be wrong here).
Coming real soon now are some very cool features that will make decisions somewhat harder: dedup+compression (from VDO) and lvmcache setup.
As for replica 3 being overkill that may be true as well but from what I've read on Ovirt and GlusterFS list archives people typically feel safer with replica 3 and run in to less disaster scenarios and can provide easier recovery. I'm not sold on Replica 3 either, Rep 3 Arbiter 1 may be more than fine but I wanted to err on the side of caution as this setup may host production servers sometime in the future.
I really wish I could get some straight answers on best configuration for Ovirt + GlusterFS but thus far it has been a big question mark. I don't know if Raid is better than JBOD and I don't know if a smaller number of SSDs would perform any better/worse than larger number of spinning disks in raid 10.
RAID is better than JBOD in terms of availability and most likely performance.
It's also more expensive and requires initial setup.
SSD will perform better than HDD.
It's also more expensive than HDD.
No one but you can provide what works best for your requirements.
Y.
