
I'm running oVirt + Gluster in HCI config and had similar questions as you when building it out.
- single point of failure in this router (not really - just in case oVirt is badly broken and I need to access internal vlans to recover it)
There is no SPOF if you're doing 3x HCI nodes. I regularly put 1 of my 3 Nodes into Maintenance or shutdown Gluster and have had no SPOFs. Are you only doing a single Node? If so, the point of failure is ... that 1 node :)
* have this router as virtual appliance inside oVirt (something like pfSense for example)
I'm running pfSense in hardware still (a Netgate ARM device). There's plenty of opinions on Reddit, StackOverflow, etc. about running any router in VM. There's several steps you'd need to take when I looked into it, and if you setup pfSense's interfaces as virtio / vhost I'd imagine you'd bump into limitations b/c those para devices weren't intended to do things like hardware offload, advanced routing, etc.; so you may have to setup PCI passthru / SR-IOV to get all of pfSense's routing capabilities. So I'm keeping pfSense in hardware ... though I've thought of creating a backup pfSense instance in VM encase of hardware disaster to keep my Internet up in "limp mode" ... but creating a cellular Hotspot is my current backup plan :)
Install all hosts and HE with public addresses
Why? The HE is a manager to the cluster and sits on the management network (ovirtmgmt), so giving it public IPs would be adding a security risk to the setup. I keep my HE accessible only via local VLAN and that's how most folks lock it down. Are you thinking the HE or HCI includes a load balancer? Eitherway, oVirt doesn't, but putting a load balancer in front of VM's and giving it your public IP would make more sense for exposing things to the Internet ... but I'm assuming too much and don't know what your cluster will be running.