Hi Yevgeny,
Thanks for your help.

No luck yet!

Q: IIUC, ovirt-engine and its host are vSphere VMs. Then, a kind of no-macspoof should be applied from the vSphere side.
A: I'm not exactly sure what setting this would be on the vmware side but we have ensured the following settings are enabled; Promiscuous Mode, MAC Address Changes, Forged Transmits.

Q: BTW, are both of them on the same vShepre host?
A: Yes, they are both on the same vsphere host.

Q: Is DHCP server another VM on that host?
A: Yes, the DHCP server is another VM on that vsphere host as well.

Q: Where/how did you "turn on Port Mirroring"?
A: I turned on Port Mirroring in the vNIC Profile called ovirtmgmt, it's off for most of my tests, however, when I turn it on, everything works.

Q: I'd start the troubleshooting by using tcpdump utility in order to pinpoint the component that blocks the traffic.
A: I have done this and I can see all DHCP messages including the DHCPACK on the bridge named 'ovirtmgmt', but when I do tcpdump on the guest VM network device 'eth0' (which is also 'vnet0' on the host)I never see the DHCPACK. This same thing happens with ARP replies from the gateway.

Here is an ACK message that is returned from the DHCP server to the 'ovirtmgmt' bridge (I don't know if this is helpful, but thought I'd include it, just in case).
16:35:22.958616 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    10.255.233.125.67 > 10.255.233.185.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x8d5c8e3d, Flags [none] (0x0000)
          Your-IP 10.255.233.185
          Client-Ethernet-Address 00:1a:4a:16:01:55
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.255.233.125
            Lease-Time Option 51, length 4: 600
            Subnet-Mask Option 1, length 4: 255.255.255.0
            BR Option 28, length 4: 10.255.233.255
            Domain-Name-Server Option 6, length 4: 10.255.233.102
            Default-Gateway Option 3, length 4: 10.255.233.1


Q: Did you try assigning a static IP instead of DHCP and then check connectivity? If that works, then the problem is on the DHCP sever side probably.
A: Yes, with a static IP on the guest, everything works fine.  It's possible, but in my naive understanding it seems unlikely, that the problem is on the DHCP server side because the ACK makes it all the way back to the bridge.  Again, I could be wrong.


Q: If you do not see any requests in the DHCP server log, then I guess, "dhclient -B" wouldn't help.
A: I see the whole DHCP process in my DHCP Server log:

Jul  5 21:21:12 mydomain dhcpd: DHCPDISCOVER from 00:1a:4a:16:01:55 via eth0
Jul  5 21:21:12 mydomain dhcpd: DHCPOFFER on 10.255.233.185 to 00:1a:4a:16:01:55 via eth0
Jul  5 21:21:12 mydomain dhcpd: DHCPREQUEST for 10.255.233.185 (10.255.233.125) from 00:1a:4a:16:01:55 via eth0
Jul  5 21:21:12 mydomain dhcpd: DHCPACK on 10.255.233.185 to 00:1a:4a:16:01:55 via eth0

In addition to "dhclient -B" allowing the guest to successfully gain an IP address from the DHCP server, I recently found that running the command "brctl ovirtmgmt setageing 0" works also.  I read that this turns the bridge into a hub, which is only a clue and not a solution, I think.  It seems that Forwarding Database changes that occur when I run that command are what allows ACK message to then pass to the guest.

Here is my Forwarding Database when the Ageing is set to 300:

01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:02:02 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev bond0 self permanent
fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent
00:50:56:8e:aa:2a dev ens32 master ovirtmgmt 
e4:d3:f1:d1:99:c4 dev ens32 master ovirtmgmt 
00:50:56:8e:d6:a2 dev ens32 master ovirtmgmt 
00:50:56:8e:d2:cd dev ens32 master ovirtmgmt 
00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent
00:50:56:8e:17:59 dev ens32 master ovirtmgmt 
fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent
00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent
00:50:56:8e:3d:f1 dev ens32 master ovirtmgmt 
e4:d3:f1:d1:99:8c dev ens32 master ovirtmgmt 
e4:d3:f1:d1:99:8b dev ens32 master ovirtmgmt 
76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent
33:33:00:00:00:01 dev vnet0 self permanent
01:00:5e:00:00:01 dev vnet0 self permanent
33:33:ff:16:01:55 dev vnet0 self permanent


Here is the Forwarding Database(fdb) after setting the Ageing to 0:

01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:02:02 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev bond0 self permanent
fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent
00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent
00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent
76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent
33:33:00:00:00:01 dev vnet0 self permanent
01:00:5e:00:00:01 dev vnet0 self permanent
33:33:ff:16:01:55 dev vnet0 self permanent

This is the result of 'brctl showmacs ovirtmgmt' with Ageing set to 300:

[root@ovirthost2 ~]# brctl showmacs ovirtmgmt
port no mac addr is local? ageing timer
  1 00:50:56:8e:17:59 no   0.10
  1 00:50:56:8e:2b:d7 no  22.30
  1 00:50:56:8e:2e:ec no   3.66
  1 00:50:56:8e:3d:f1 no   1.16
  1 00:50:56:8e:a4:66 no  15.34
  1 00:50:56:8e:a4:cc no  11.39
  1 00:50:56:8e:aa:2a no   3.66
  1 00:50:56:8e:be:ca yes   0.00
  1 00:50:56:8e:be:ca yes   0.00
  1 00:50:56:8e:d2:cd no   0.10
  1 00:50:56:8e:d6:a2 no   3.70
  1 e4:d3:f1:d1:99:8b no   0.26
  1 e4:d3:f1:d1:99:8c no   0.26
  1 e4:d3:f1:d1:99:c4 no   0.00
  2 fe:1a:4a:16:01:55 yes   0.00
  2 fe:1a:4a:16:01:55 yes   0.00

And here is the result of 'brctl showmacs ovirtmgmt' with Ageing set to 0:

[root@ovirthost2 ~]# brctl showmacs ovirtmgmt
port no mac addr is local? ageing timer
  1 00:50:56:8e:be:ca yes   0.00
  1 00:50:56:8e:be:ca yes   0.00
  2 fe:1a:4a:16:01:55 yes   0.00
  2 fe:1a:4a:16:01:55 yes   0.00

Maybe this will give more clues as to what's going on.


Q: Please turn iptables/firewalld off.
A: I have tried this ('service iptables stop') and it doesn't seem to make a difference.


Thanks again for your help,
Clint










On Mon, Jul 4, 2016 at 2:41 AM, Yevgeny Zaspitsky <yzaspits@redhat.com> wrote:
Adding mailing list back...

On Mon, Jul 4, 2016 at 3:38 PM, Yevgeny Zaspitsky <yzaspits@redhat.com> wrote:
Clint,

Sorry, I missed that you already tried that.

Here are my thoughts (some more shooting in the dark) after reading your description again:
  • You have quite complicate setup. IIUC, ovirt-engine and its host are vSphere VMs. Then, a kind of no-macspoof should be applied from the vSphere side. BTW, are both of them on the same vShepre host? Is DHCP server another VM on that host?
  • Where/how did you "turn on Port Mirroring"?
  • I'd start the troubleshooting by using tcpdump utility in order to pinpoint the component that blocks the traffic.
  • Did you try assigning a static IP instead of DHCP and then check connectivity? If that works, then the problem is on the DHCP sever side probably.
  • If you do not see any requests in the DHCP server log, then I guess, "dhclient -B" wouldn't help.
  • Please turn iptables/firewalld off.

Regards,
Yevgeny

On Sun, Jul 3, 2016 at 9:06 PM, Yevgeny Zaspitsky <yzaspits@redhat.com> wrote:
Hello,

IIUC using vdsm macspoof hook would help - reading [1] should help you configuring that.

[1] https://github.com/oVirt/vdsm/blob/master/vdsm_hooks/macspoof/README

Hope that helps,
Yevgeny

On Thu, Jun 30, 2016 at 6:11 AM, Clint Smith <clint.smith.maui@gmail.com> wrote:
Hello,

I have been experimenting with oVirt for the last couple of weeks and I must say it has a lot of nice features.  I really like it, however, I am having a heck of a time getting the guest networking all set up correctly. I am hoping that someone can give me a little guidance in figuring this out.  I apologize in advance if some of my terminology is off,  I am new.

Here is a brief intro to my setup:
I created a Centos 7 VM within a vSphere/ESXI environment and then installed ovirt-engine on it.  I also created another Centos 7 VM and set it up as a host.  I have configured the Cluster and Host via the oVirt Administration Portal.  For simplicity, I am using the default ovirtmgmt network as my only logical network, however I have tried several different schemes with no luck.  I have a DHCP server and a DNS server that are siblings to the oVirt host and the engine.  Both the engine and the host have been upgraded to version 4.0.

The problem:
My thought was that I would have the guest VMs on the oVirt host use my existing DHCP server to get their IP addresses, at least at first.  The problem I am having is that the DHCPACK is not making it back across the ovirtmgmt bridge and on to the guest.  If I tell dhclient(from the guest) to force a Broadcast (by using the –B option) on the DHCP server, it will work.  This is not a solution, just a clue.  Another clue is that ARP replies from the gateway don’t make it back to the machine, preventing pings even when I force the IP.  Lastly, If I turn on Port Mirroring, everything works fine, but it’s my understanding that this is only for debugging purposes.

What I have tried (in no particular order):
  • Reading the docs
  • Turning on VLAN tagging.
  • Installing the mac-spoofing hook, making the configuration changes to the engine, and then turning it on in the VM config.  I also verified that the ‘filterref’ tag was removed using virsh.
  • Setting up a second logical network on a different subnet, and connecting it to an additional network interface that I added to the host.  On the host, I setup dnsmasq as a DNS and DHCP server.  I got this working up to the point of having the same issues that I was having using the existing DHCP and DNS servers on the ovirtmgmt network.
  • I have tried various changes to iptables as well as the original settings as well as verified that ebtables is not blocking any traffic.  I did configure iptables for logging and noticed it was dropping some traffic related to DHCP, however it seemed like it was DISCOVER or REQUEST traffic due to the IN, OUT, SRC, and DST variables in the log.  I have viewed the DHCP server logs multiple times and I can see that it is receiving the DISCOVER and the REQUEST from my guests MAC and sending the OFFER and ACK consistently.
  • Setting SELinux to Permissive
  • Setting ip_forward to 1
  • Turning STP ON on the bridge
  • Changing the bridge delay
  • Setting up a dhcrelay using dnsmasq (not sure I implemented this right though)
I am really shooting in the dark when it comes to networking because I am learning a lot of this on the fly.   I feel like I must have a misconception about how networking should work with oVirt.  Is my entire approach naïve? Any help/guidance that someone could offer would be much appreciated.

Thanks,
Clint




_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users