It's better when you post distinct problems in distinct posts.
I'll answer on the CPU aspect, which may not be related to the networking topic at
all.
Sounds like you're adding Haswell parts to a farm that was built on Skylakes. In order
for VMs to remain mobile across hosts, oVirt needs to establish a baseline of
CPU/instruction set features, that VMs will be let to see, so an application/VM launched
on a machine with say AVX512 support won't all of a sudden find itself on a host that
doesn't support that (please note that all the various Spectre mitigations resulted in
distinct CPU subtypes depending on microcode patches being applied to hosts or not).
That's the reason it says in the documentation, that the initial setup should be done
on the "oldest" machine, to ensure the proper baseline.
In case you add older hardware later, you can lower the base machine type (within some
limits) by changing the CPU type in the appropriate cluster.
I do believe it requires restarting the management engine to pick up the change.
Of course, if you want to actually use those nice new instructions, you'll have to
divide your hosts into distinct clusters.
With regards to the 'ovirt-hosted-engine-setup' playboook being use to add hosts,
that may just be misleading naming. Rest assured it won't try to add a second
management engine: that is too complicated to fit into a single playbook.