I had similar confusion, and trying to assign static IPs was just not working for me.
Here is how I got it to work:
#1 you absolutely require a DNS server you can add entries to. I tried to go without, and it just does not work.
For me, management network is on vlan 10, so each host grabs a 172.16.10.xx address. The oVirt engine (self hosted) also grabs an address in this subnet. I look in the DHCP leases and turn those into static assignments in the router, but left the config on the hosts to DHCP. Add those IPs to your DNS so that you can reach each host, and the engine, at
hostname.domain.net (or whatever your fully qualified domain name is for each)
Now, set your VM network to DHCP on a different, specific VLAN. Set that vlan up in your router with a small DHCP pool, only big enough for the VMs you want to run. Make sure you allow that vlan through to the switchports connected to the nic you are using for your VMs.
For me, the VMs are on vlan 30, so the subnet is 172.16.30.xx. Each VM boots up, makes a DHCP request on vlan 30, and is assigned an address. Then again go into your router and make this a static entry, leaving the config in the VM to DHCP. Once you install the agent in the VM, the IP will show up in the engine GUI. Then you can assign an FQDN to each vm's IP I'm your DNS server.
Set up this way, I can migrate the VM to whichever host, and still access it over RDP by it's FQDN.
I also have dual NICs but I set mine up as a team (bridge) instead of one for MGMT and one for VMs. That way, I can also unplug anyone Ethernet cable from any machine, and nothing happens, both the MGMT and VM network keep ticking away happily. I have each NIC plugged into a separate switch chassis, to gain redundancy on the switches.
Hope some part of this helps you!