VMs don't receive unicast packets, can't reach gateway, don't get ARP responses.

Hey All, I'm reaching out because I'm working on a PoC ovirt setup, and running into some networking issues from my VMs that I just have not been able to nail down or figure out. Setup: Three Hypervisors running on the ovirt image, one NIC on each is connected to a trunk port on our data network, the other is connected to our storage network. I'm just running a basic test setup with three logical networks - ovirtmgmt, stroage, vm_network the vNics have no network filtering enabled, I have disabled and stopped firewalld on the hypervisors as well. ovirtmgmt and storage have IPs on the hypervisor, vm_network does not have an IP (Thought I did try it with no difference.) I currently just have three VMs running since it's a PoC, one on each host - they are all connected to the vm_network, and all are unable to reach their gateway (a linux device), and unable to reach each other unless they are on the same hypervisor - this is the main issue I'm dealing with. Investigation done so far: Running a ping from the VM to the Gateway IP and capturing data along the flow gives me this: If I run a tcpdump on the hypervisor I can see ARP requests being broadcasted on the correct interface (vm_network, vnet0) from the VM looking for the gateway, however I never see any ARP response from the gateway on the hypervisor, and the VMs never update their arp table and never actually attempt to send any ICMP packets. If I run a tcpdump on the gateway device, I can see ARP requests from the VM coming in, and I can see the gateway giving a unicast ARP reply - again I never see that arp reply on the hypervisor, and I never see it reach the VM. Now where it gets a little bit interesting at least. if I run an arping from the gateway device to the VM - I get a response from the VM on the broadcast arp, but I do not get any responses (or see any traffic on the hypervisor) when it moves to a unicast arp probe ping. After doing this the VM will also update it's ARP table with the gateway. If I attempt to ping the gateway from the VM after this, it will send ICMP packets, which do reach the gateway, the gateway does respond... but just like all other unicast things I never see that response on the hypervisor, and it never reaches the VM. (This only lasts for a short period of time until the VM fails to get a new ARP response and removes the entry from the arp table.) There is no firewall setup on the gateway that would prevent any of this traffic, and everything with that gateway, vlan, and IPs work just fine outside of ovirt. The behavior between VMs (on different hypervisors) is exactly the same as VM to Gateway. Where arp requests go out, the VM on a different hypervisor will see and respond to the arp broadcast request, but I never see that unicast ARP reply make it to a tcpdump on the hypervisor. TLDR: Broadcast traffic reaches the VMs, unicast packets from the VM reach their destination (If they have a record for them in their arp table), unicast replies and unicast requests to the VMs never even show up in a tcpdump on the hypervisor. No firewalls in place to explain any of these issues. I am sure I likely screwed something up during the configuration of this, but I don't see my mistake anywhere

Just to update - figured this one out, was my own fault for attemping the PoC in a vmware environment, which did not treat layer 2/mac address stuff like I thought it did. Once I put two and two together, and setup some mac spoofing, things all worked as expected.
participants (1)
-
jeff.parrish@dentsu.com