Hoping someone can help with a problem my team is seeing under oVirt. We are making heavy use of macvlan interfaces (in VEPA mode) on-top of virtual functions, under VMs being managed by oVirt. In this scenario IPv6 is not playing nicely, with no traffic going through, and messages about neighbor solicitation. There are some pointers out there indicating the issue stems from the fact IPv6 utilizing multicast for neighbor solicitation, but nothing we have tried seems to work around this issue. The problem is made all the most bizarre by the fact that on the same hardware libvirt + virt-manager works perfectly fine. I have looked at the corresponding xml for the guests, and nothing seems to point to the underlying cause for oVirt to fail, but libvirt to succeed. * Intel XL710 40G NICs (i40e/i40evf drivers) * CentOS 7.1 (fully up-to-date) * oVirt 3.6 * libvirt-1.2.17 * virt-manager-1.2.1-8 Thanks for any pointers folks can provide. - jkt

----- Original Message ----- | From: "Jay Turner" <jkt(a)iix.net&gt; | To: users(a)ovirt.org | Sent: Tuesday, February 23, 2016 5:13:21 PM | Subject: Re: [ovirt-users] macvlan + IPv6 | | As a follow-up to this, I made some headway in sorting out the source of the | issue, but hoping someone can give me a pointer to where this is happening | in the code, as well as some understanding for why. | | In oVirt, when I allocate a virtual function to a guest, a new MAC address is | generated for the VF (as it should be) from the MAC address pool in oVirt, | and then that MAC address is written to the VF on the hypervisor. Thus I end | up with something like: | | : ens11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master i40e | state UP mode DEFAULT qlen 1000 | link/ether 3c:fd:fe:9d:a1:38 brd ff:ff:ff:ff:ff:ff | vf 0 MAC 00:1a:4a:16:01:52, spoof checking on, link-state auto | | This *is not* how it happens under libvirt/virt-manager, however. When | allocating a VF to a guest under libvirt, a random MAC address is generated | and associated with the VF under the guest, but it is not written back to | the hypervisor, and is instead left as 00:00:00:00:00:00. | | I am pretty sure this writing of the MAC address at the hypervisor is causing | at least some of the issues I'm seeing, as with the Intel cards, that | prevents the guest from changing/adding a new MAC address, which is what | happens with the instantiation of a macvlan interface. | | So can anyone point me to where in the oVirt code this MAC address assignment | is occurring? Also curious why oVirt does this assignment, but libvirt does | not. | | Thanks! | | - jkt | | On Mon, Feb 22, 2016 at 2:51 PM Jay Turner < jkt(a)iix.net > wrote: | | | | Hoping someone can help with a problem my team is seeing under oVirt. | | We are making heavy use of macvlan interfaces (in VEPA mode) on-top of | virtual functions, under VMs being managed by oVirt. In this scenario IPv6 | is not playing nicely, with no traffic going through, and messages about | neighbor solicitation. There are some pointers out there indicating the | issue stems from the fact IPv6 utilizing multicast for neighbor | solicitation, but nothing we have tried seems to work around this issue. | | The problem is made all the most bizarre by the fact that on the same | hardware libvirt + virt-manager works perfectly fine. I have looked at the | corresponding xml for the guests, and nothing seems to point to the | underlying cause for oVirt to fail, but libvirt to succeed. | | * Intel XL710 40G NICs (i40e/i40evf drivers) | * CentOS 7.1 (fully up-to-date) | * oVirt 3.6 | * libvirt-1.2.17 | * virt-manager-1.2.1-8 | | Thanks for any pointers folks can provide. | | - jkt | | _______________________________________________ | Users mailing list | Users(a)ovirt.org | http://lists.ovirt.org/mailman/listinfo/users | Could you please provide the dumpxml for both setups (oVirt and non-oVirt)? (sudo virsh -r dumpxml <domain>) Thanks, Edy.

As a follow-up to this, I made some headway in sorting out the source of the issue, but hoping someone can give me a pointer to where this is happening in the code, as well as some understanding for why. In oVirt, when I allocate a virtual function to a guest, a new MAC address is generated for the VF (as it should be) from the MAC address pool in oVirt, and then that MAC address is written to the VF on the hypervisor. Thus I end up with something like: : ens11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master i40e state UP mode DEFAULT qlen 1000 link/ether 3c:fd:fe:9d:a1:38 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:1a:4a:16:01:52, spoof checking on, link-state auto This *is not* how it happens under libvirt/virt-manager, however. When allocating a VF to a guest under libvirt, a random MAC address is generated and associated with the VF under the guest, but it is not written back to the hypervisor, and is instead left as 00:00:00:00:00:00. I am pretty sure this writing of the MAC address at the hypervisor is causing at least some of the issues I'm seeing, as with the Intel cards, that prevents the guest from changing/adding a new MAC address, which is what happens with the instantiation of a macvlan interface. So can anyone point me to where in the oVirt code this MAC address assignment is occurring?

Also curious why oVirt does this assignment, but libvirt does not.

Attached. macvlan-1.xml is from oVirt (and includes the VF MAC address) macvlan-test-2.xml is from virt-manager (and does not include the VF MAC address)

<domain type='kvm' id='2'>

<interface type='bridge'> <mac address='00:1a:4a:16:01:51'/> <source bridge='ovirtmgmt'/> <target dev='vnet0'/> <model type='virtio'/> <filterref filter='vdsm-no-mac-spoofing'/> <link state='up'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='hostdev'> <mac address='00:1a:4a:16:01:52'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x1'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </interface>

</domain>

<domain type='kvm' id='12'>

<interface type='network'> <mac address='52:54:00:22:54:ba'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>

<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev>

</domain>