Howto: Create a new network profile that doesn't have the MAC spoofing filter
included. In my case I used one without any filters, you may want to be more careful.
Background:
I had tried the nested approach with the default settings and found that the hosted-engine
setup worked just find to the point where it was running as a nested VM on the install
node, but that it failed to tie in the other gluster nodes or to communicate to the
outside: There was something wrong with the network, but no obivous failures.
I tried using other, simpler "client hypervisors" (it's all KVM anyway)
VirtualBox and 'native' KVM and the symptoms were the same: The VMs ran just fine,
but the network didn't connect.
And then I remembered that obviously a Hypervisor need to override the MAC for each client
VM if you want full network access and I had come across "no-mac-spoofing" often
enough it somehow stayed on my mind... and finally it clicked.
Anyhow, validing full functionality now and the main interest in this is the ability to do
a full simulation/testing of oVirt 4.3->4.4 upgrades on 3-node HCI setups, which
I'd rather not do on the live physical system.
Of course, it's also quite cool!