I currently only have two IPs assigned to me...I can try and take another, but that may not route out of the rack.  I've got the VM on one of the IPs and the host on the other currently.

The switch is a "web-managed" basic 8-port switch (thrown in for testing while the "real" switch is in transit).  It has the 3 ports the hosts are plugged in configured with vlan 1 untagged, set as PVID, and vlan 2 tagged.  Another port on the switch is untagged on vlan 1 connected to the router for the ovirtmgmt network (protected by a VPN, but not "burning" public IPs for mgmt purposes), another couple ports are untagged on vlan 2.  One of those ports goes out of the rack, another goes to the router's internet port.  Router gets to the internet just fine.

VM: 

kusznir@FusionPBX:~$ ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 00:1a:4a:16:01:51 brd ff:ff:ff:ff:ff:ff

    inet 162.248.147.31/24 brd 162.248.147.255 scope global eth0

       valid_lft forever preferred_lft forever

    inet6 fe80::21a:4aff:fe16:151/64 scope link 

       valid_lft forever preferred_lft forever

kusznir@FusionPBX:~$ ip route

default via 162.248.147.1 dev eth0 

162.248.147.0/24 dev eth0  proto kernel  scope link  src 162.248.147.31 

kusznir@FusionPBX:~$ 

Host:

[root@ovirt3 ~]# ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP qlen 1000

    link/ether 00:21:9b:98:2f:44 brd ff:ff:ff:ff:ff:ff

3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000

    link/ether 00:21:9b:98:2f:46 brd ff:ff:ff:ff:ff:ff

4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000

    link/ether 00:21:9b:98:2f:48 brd ff:ff:ff:ff:ff:ff

5: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000

    link/ether 00:21:9b:98:2f:4a brd ff:ff:ff:ff:ff:ff

6: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 

    link/ether 8e:1b:51:60:87:55 brd ff:ff:ff:ff:ff:ff

7: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 

    link/ether 00:21:9b:98:2f:44 brd ff:ff:ff:ff:ff:ff

    inet 192.168.8.13/24 brd 192.168.8.255 scope global dynamic ovirtmgmt

       valid_lft 54830sec preferred_lft 54830sec

11: em1.2@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master Public_Cable state UP 

    link/ether 00:21:9b:98:2f:44 brd ff:ff:ff:ff:ff:ff

12: Public_Cable: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 

    link/ether 00:21:9b:98:2f:44 brd ff:ff:ff:ff:ff:ff

    inet 162.248.147.33/24 brd 162.248.147.255 scope global Public_Cable

       valid_lft forever preferred_lft forever

14: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UNKNOWN qlen 500

    link/ether fe:1a:4a:16:01:54 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::fc1a:4aff:fe16:154/64 scope link 

       valid_lft forever preferred_lft forever

15: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UNKNOWN qlen 500

    link/ether fe:1a:4a:16:01:52 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::fc1a:4aff:fe16:152/64 scope link 

       valid_lft forever preferred_lft forever

16: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UNKNOWN qlen 500

    link/ether fe:1a:4a:16:01:53 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::fc1a:4aff:fe16:153/64 scope link 

       valid_lft forever preferred_lft forever

17: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master Public_Cable state UNKNOWN qlen 500

    link/ether fe:1a:4a:16:01:51 brd ff:ff:ff:ff:ff:ff

    inet6 fe80::fc1a:4aff:fe16:151/64 scope link 

       valid_lft forever preferred_lft forever

[root@ovirt3 ~]# ip route

default via 192.168.8.1 dev ovirtmgmt 

162.248.147.0/24 dev Public_Cable  proto kernel  scope link  src 162.248.147.33 

169.254.0.0/16 dev ovirtmgmt  scope link  metric 1007 

169.254.0.0/16 dev Public_Cable  scope link  metric 1012 

192.168.8.0/24 dev ovirtmgmt  proto kernel  scope link  src 192.168.8.13 

[root@ovirt3 ~]# brctl show

bridge name bridge id STP enabled interfaces

;vdsmdummy; 8000.000000000000 no

Public_Cable 8000.00219b982f44 no em1.2

vnet3

ovirtmgmt 8000.00219b982f44 no em1

vnet0

vnet1

vnet2

[root@ovirt3 ~]# 

I did see that the cluster settings has a switch type setting; currently at the default "LEGACY", it also has "OVS" as an option.  Not sure if that matters or not.

I configured another VM on the network, and static'ed an IP, and could ping the other VM as well as the host, but not the internet.  The host can still ping the internet.

--Jim