10:53 p.m.
Hey All,
I'm reaching out because I'm working on a PoC ovirt setup, and running into some
networking issues from my VMs that I just have not been able to nail down or figure out.
Setup:
Three Hypervisors running on the ovirt image, one NIC on each is connected to a trunk port
on our data network, the other is connected to our storage network.
I'm just running a basic test setup with three logical networks - ovirtmgmt, stroage,
vm_network the vNics have no network filtering enabled, I have disabled and stopped
firewalld on the hypervisors as well. ovirtmgmt and storage have IPs on the hypervisor,
vm_network does not have an IP (Thought I did try it with no difference.)
I currently just have three VMs running since it's a PoC, one on each host - they are
all connected to the vm_network, and all are unable to reach their gateway (a linux
device), and unable to reach each other unless they are on the same hypervisor - this is
the main issue I'm dealing with.
Investigation done so far:
Running a ping from the VM to the Gateway IP and capturing data along the flow gives me
this:
If I run a tcpdump on the hypervisor I can see ARP requests being broadcasted on the
correct interface (vm_network, vnet0) from the VM looking for the gateway, however I never
see any ARP response from the gateway on the hypervisor, and the VMs never update their
arp table and never actually attempt to send any ICMP packets.
If I run a tcpdump on the gateway device, I can see ARP requests from the VM coming in,
and I can see the gateway giving a unicast ARP reply - again I never see that arp reply on
the hypervisor, and I never see it reach the VM.
Now where it gets a little bit interesting at least.
if I run an arping from the gateway device to the VM - I get a response from the VM on the
broadcast arp, but I do not get any responses (or see any traffic on the hypervisor) when
it moves to a unicast arp probe ping. After doing this the VM will also update it's
ARP table with the gateway. If I attempt to ping the gateway from the VM after this, it
will send ICMP packets, which do reach the gateway, the gateway does respond... but just
like all other unicast things I never see that response on the hypervisor, and it never
reaches the VM. (This only lasts for a short period of time until the VM fails to get a
new ARP response and removes the entry from the arp table.)
There is no firewall setup on the gateway that would prevent any of this traffic, and
everything with that gateway, vlan, and IPs work just fine outside of ovirt.
The behavior between VMs (on different hypervisors) is exactly the same as VM to Gateway.
Where arp requests go out, the VM on a different hypervisor will see and respond to the
arp broadcast request, but I never see that unicast ARP reply make it to a tcpdump on the
hypervisor.
TLDR:
Broadcast traffic reaches the VMs, unicast packets from the VM reach their destination (If
they have a record for them in their arp table), unicast replies and unicast requests to
the VMs never even show up in a tcpdump on the hypervisor. No firewalls in place to
explain any of these issues.
I am sure I likely screwed something up during the configuration of this, but I don't
see my mistake anywhere