vlan-tagging on non-tagged network
Hi everybody! I am experiencing a behaviour of ovirt, of which I don't know whether it is expected or not. My setup is as follows: A virtual machine has a logical network attached to it, which is configured without vlan-tagging and listens to the name 'trunk'. The VM is running an openvpn server. It is a patched openvpn version, including vlan-tagging. That is, openvpn clients get a vlan tag. This should not really be an issue but should satisfy the "why do you want to do it in the first place"-questions. Anyhow, effectively, the VM simply puts vlan-tagged ethernet-frames on the virtual network. These frames, however, never make it to the host's network bridge, which represents the logical network. My observations are: According to tcpdump, the vlan-tagged packages arrive at the "eth1"-interface inside the VM (which *is* the correct interface). Again, according to tcpdump, these packages never arrive at the corresponding network-bridge (i.e., the interface 'trunk') on the host. I know that the setup itself is feasible with KVM---I have it working on a proxmox-machine. Therefore, my conclusion is, that ovirt doesn't like vlan-tagged ethernet-frames on non-tagged logical networks, and somehow filters them out, though I don't really see on what "level" that would happen (Handling the ethernet frames should be a concern of KVM/QEMU/Linux only, once ovirt has started the VM). So this problem could be a CentOS issue, but I really don't see why CentOS should act differently than debian does (proxmox is debian-based). Is this a known/wanted/expected behaviour of ovirt, and can I somehow prevent or elude it? Any help is much appreciated! Of course I am happy to provide more information if that helps helping me :) Regards, Felix
Hey Felix. IIUC your frames are dropped by the bridge. Ovirt uses Linux Bridges To connect virtual machines to 'networks'. The guest connects to the bridge using a tap device which usually is called 'vnet<number>'. So, just to verify, can you please tcpdump both on the bridge device and on the tap device? The bridge can be quite noisy so I suggest filtering traffic using the guest's MAC address. So I am not sure what protocol you use for tunneling but applying a filter similar to this one should do the job: tcpdump -n -i vnet0 -vvv -s 1500 'udp[38:4]=0x001a4aaeec8e' My guess is that you will observe traffic on the tap device, but not on the bridge. You didn't specify which centOS version you use but I do remember seeing people complaining about Linux bridges discarding their tagged frames. You can -maybe- also observe the 'dropped' counter increases on the bridge by running: 'ip -s link show dev trunk' There were a few bugs on rhel6/7 about this, specifically I remember https://bugzilla.redhat.com/show_bug.cgi?id=1174291 and https://bugzilla.redhat.com/show_bug.cgi?id=1200275#c20 Also, is the vlan module loaded on your host? 'lsmod |grep 8021q' Thanks, Ido ----- Original Message ----- From: "Felix Pepinghege" <pepinghege@ira.uka.de> To: Users@ovirt.org Sent: Monday, June 15, 2015 11:33:39 AM Subject: [ovirt-users] vlan-tagging on non-tagged network Hi everybody! I am experiencing a behaviour of ovirt, of which I don't know whether it is expected or not. My setup is as follows: A virtual machine has a logical network attached to it, which is configured without vlan-tagging and listens to the name 'trunk'. The VM is running an openvpn server. It is a patched openvpn version, including vlan-tagging. That is, openvpn clients get a vlan tag. This should not really be an issue but should satisfy the "why do you want to do it in the first place"-questions. Anyhow, effectively, the VM simply puts vlan-tagged ethernet-frames on the virtual network. These frames, however, never make it to the host's network bridge, which represents the logical network. My observations are: According to tcpdump, the vlan-tagged packages arrive at the "eth1"-interface inside the VM (which *is* the correct interface). Again, according to tcpdump, these packages never arrive at the corresponding network-bridge (i.e., the interface 'trunk') on the host. I know that the setup itself is feasible with KVM---I have it working on a proxmox-machine. Therefore, my conclusion is, that ovirt doesn't like vlan-tagged ethernet-frames on non-tagged logical networks, and somehow filters them out, though I don't really see on what "level" that would happen (Handling the ethernet frames should be a concern of KVM/QEMU/Linux only, once ovirt has started the VM). So this problem could be a CentOS issue, but I really don't see why CentOS should act differently than debian does (proxmox is debian-based). Is this a known/wanted/expected behaviour of ovirt, and can I somehow prevent or elude it? Any help is much appreciated! Of course I am happy to provide more information if that helps helping me :) Regards, Felix _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Ido, hi everybody, sorry that I kept you waiting for two months, I only just found the time to go back to this problem. You were completely right with your guess. The ethernet frames do appear on the vnet-interface, but not on the bridge. The dropped-counter seems to be independent from these losses, though. However, while this tells me *where* the problem is, I still don't know *what* the problem is. I've done some research, but couldn't find anything particularly helpful. An interesting point may be that this problem is mono-directional. That is, the bridge happily passes vlan-tagged frames from the ethernet device to the vnet, but not the other way around. Untagged ethernet frames make their way through the brigde, no matter where they come from. The vlan module is loaded, as to the versioning questions: # cat /etc/centos-release ; uname -s -v -r CentOS Linux release 7.1.1503 (Core) Linux 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 The guest OS is an up-to-date Debian Jessie, which should not matter, though, as the frames get lost on the doorstep of the bridge on the host. Again, any suggestions are much appreciated! Regards, Felix Am 16.06.2015 um 08:27 schrieb Ido Barkan:
Hey Felix.
IIUC your frames are dropped by the bridge. Ovirt uses Linux Bridges To connect virtual machines to 'networks'. The guest connects to the bridge using a tap device which usually is called 'vnet<number>'.
So, just to verify, can you please tcpdump both on the bridge device and on the tap device? The bridge can be quite noisy so I suggest filtering traffic using the guest's MAC address. So I am not sure what protocol you use for tunneling but applying a filter similar to this one should do the job: tcpdump -n -i vnet0 -vvv -s 1500 'udp[38:4]=0x001a4aaeec8e'
My guess is that you will observe traffic on the tap device, but not on the bridge. You didn't specify which centOS version you use but I do remember seeing people complaining about Linux bridges discarding their tagged frames. You can -maybe- also observe the 'dropped' counter increases on the bridge by running: 'ip -s link show dev trunk'
There were a few bugs on rhel6/7 about this, specifically I remember https://bugzilla.redhat.com/show_bug.cgi?id=1174291 and https://bugzilla.redhat.com/show_bug.cgi?id=1200275#c20
Also, is the vlan module loaded on your host? 'lsmod |grep 8021q'
Thanks, Ido
----- Original Message ----- From: "Felix Pepinghege" <pepinghege@ira.uka.de> To: Users@ovirt.org Sent: Monday, June 15, 2015 11:33:39 AM Subject: [ovirt-users] vlan-tagging on non-tagged network
Hi everybody!
I am experiencing a behaviour of ovirt, of which I don't know whether it is expected or not. My setup is as follows: A virtual machine has a logical network attached to it, which is configured without vlan-tagging and listens to the name 'trunk'. The VM is running an openvpn server. It is a patched openvpn version, including vlan-tagging. That is, openvpn clients get a vlan tag. This should not really be an issue but should satisfy the "why do you want to do it in the first place"-questions. Anyhow, effectively, the VM simply puts vlan-tagged ethernet-frames on the virtual network. These frames, however, never make it to the host's network bridge, which represents the logical network. My observations are: According to tcpdump, the vlan-tagged packages arrive at the "eth1"-interface inside the VM (which *is* the correct interface). Again, according to tcpdump, these packages never arrive at the corresponding network-bridge (i.e., the interface 'trunk') on the host. I know that the setup itself is feasible with KVM---I have it working on a proxmox-machine. Therefore, my conclusion is, that ovirt doesn't like vlan-tagged ethernet-frames on non-tagged logical networks, and somehow filters them out, though I don't really see on what "level" that would happen (Handling the ethernet frames should be a concern of KVM/QEMU/Linux only, once ovirt has started the VM). So this problem could be a CentOS issue, but I really don't see why CentOS should act differently than debian does (proxmox is debian-based). Is this a known/wanted/expected behaviour of ovirt, and can I somehow prevent or elude it?
Any help is much appreciated! Of course I am happy to provide more information if that helps helping me :)
Regards, Felix _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi again, I just found the reason for the loss of the packages. Either ovirt or centos install some ebtables rules in the "nat" table. These rules filter ARP packages coming from vnet interfaces. Flushing the table solved my problems. However, I have no idea what the purpose of these rules is, so there might be unwanted side-effects to just flushing the rules away.. But for now I am happy! Thanks to Ido for your help! Regards, Felix Am 17.08.2015 um 13:20 schrieb Felix Pepinghege:
Hi Ido, hi everybody,
sorry that I kept you waiting for two months, I only just found the time to go back to this problem.
You were completely right with your guess. The ethernet frames do appear on the vnet-interface, but not on the bridge. The dropped-counter seems to be independent from these losses, though.
However, while this tells me *where* the problem is, I still don't know *what* the problem is. I've done some research, but couldn't find anything particularly helpful. An interesting point may be that this problem is mono-directional. That is, the bridge happily passes vlan-tagged frames from the ethernet device to the vnet, but not the other way around. Untagged ethernet frames make their way through the brigde, no matter where they come from.
The vlan module is loaded, as to the versioning questions: # cat /etc/centos-release ; uname -s -v -r CentOS Linux release 7.1.1503 (Core) Linux 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015
The guest OS is an up-to-date Debian Jessie, which should not matter, though, as the frames get lost on the doorstep of the bridge on the host.
Again, any suggestions are much appreciated!
Regards, Felix
Am 16.06.2015 um 08:27 schrieb Ido Barkan:
Hey Felix.
IIUC your frames are dropped by the bridge. Ovirt uses Linux Bridges To connect virtual machines to 'networks'. The guest connects to the bridge using a tap device which usually is called 'vnet<number>'.
So, just to verify, can you please tcpdump both on the bridge device and on the tap device? The bridge can be quite noisy so I suggest filtering traffic using the guest's MAC address. So I am not sure what protocol you use for tunneling but applying a filter similar to this one should do the job: tcpdump -n -i vnet0 -vvv -s 1500 'udp[38:4]=0x001a4aaeec8e'
My guess is that you will observe traffic on the tap device, but not on the bridge. You didn't specify which centOS version you use but I do remember seeing people complaining about Linux bridges discarding their tagged frames. You can -maybe- also observe the 'dropped' counter increases on the bridge by running: 'ip -s link show dev trunk'
There were a few bugs on rhel6/7 about this, specifically I remember https://bugzilla.redhat.com/show_bug.cgi?id=1174291 and https://bugzilla.redhat.com/show_bug.cgi?id=1200275#c20
Also, is the vlan module loaded on your host? 'lsmod |grep 8021q'
Thanks, Ido
----- Original Message ----- From: "Felix Pepinghege" <pepinghege@ira.uka.de> To: Users@ovirt.org Sent: Monday, June 15, 2015 11:33:39 AM Subject: [ovirt-users] vlan-tagging on non-tagged network
Hi everybody!
I am experiencing a behaviour of ovirt, of which I don't know whether it is expected or not. My setup is as follows: A virtual machine has a logical network attached to it, which is configured without vlan-tagging and listens to the name 'trunk'. The VM is running an openvpn server. It is a patched openvpn version, including vlan-tagging. That is, openvpn clients get a vlan tag. This should not really be an issue but should satisfy the "why do you want to do it in the first place"-questions. Anyhow, effectively, the VM simply puts vlan-tagged ethernet-frames on the virtual network. These frames, however, never make it to the host's network bridge, which represents the logical network. My observations are: According to tcpdump, the vlan-tagged packages arrive at the "eth1"-interface inside the VM (which *is* the correct interface). Again, according to tcpdump, these packages never arrive at the corresponding network-bridge (i.e., the interface 'trunk') on the host. I know that the setup itself is feasible with KVM---I have it working on a proxmox-machine. Therefore, my conclusion is, that ovirt doesn't like vlan-tagged ethernet-frames on non-tagged logical networks, and somehow filters them out, though I don't really see on what "level" that would happen (Handling the ethernet frames should be a concern of KVM/QEMU/Linux only, once ovirt has started the VM). So this problem could be a CentOS issue, but I really don't see why CentOS should act differently than debian does (proxmox is debian-based). Is this a known/wanted/expected behaviour of ovirt, and can I somehow prevent or elude it?
Any help is much appreciated! Of course I am happy to provide more information if that helps helping me :)
Regards, Felix _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Aug 17, 2015 at 02:05:12PM +0200, Felix Pepinghege wrote:
Hi again,
I just found the reason for the loss of the packages. Either ovirt or centos install some ebtables rules in the "nat" table. These rules filter ARP packages coming from vnet interfaces. Flushing the table solved my problems. However, I have no idea what the purpose of these rules is, so there might be unwanted side-effects to just flushing the rules away.. But for now I am happy!
Can you share your ebtables before and after flushing? How did you flush the faulty rules? Would the proble reappear after next reboot? Dan.
Hi Dan, I already wrote a new mail to the list in which I elaborate the situation. This should answer the last two of your questions (if not, I'm happy to provide more information, if I can). Unfortunately I didn't save the ebtables output that showed the rules I ran into, so I don't know them exactly. But I remember them explicitly filtering all ARP-packets comming from the vnet with a different mac address than the registered one. However, just running '# ebtables -t nat -L' on a host with existing vnets should give you the rules, in case you haven't change any defaults of the engine.. Regards, Felix On 08/18/2015 11:21 AM, Dan Kenigsberg wrote:
On Mon, Aug 17, 2015 at 02:05:12PM +0200, Felix Pepinghege wrote:
Hi again,
I just found the reason for the loss of the packages. Either ovirt or centos install some ebtables rules in the "nat" table. These rules filter ARP packages coming from vnet interfaces. Flushing the table solved my problems. However, I have no idea what the purpose of these rules is, so there might be unwanted side-effects to just flushing the rules away.. But for now I am happy!
Can you share your ebtables before and after flushing? How did you flush the faulty rules? Would the proble reappear after next reboot?
Dan.
Sorry for flooding the mailing list, but I have some new insights in why these things I described happen. So just in case someone stumbles over it in the future, I like to elaborate. The problem is my specific use-case, that is, the VM being an openVPN server. By default, ovirt expects exactly one mac address per VM. As one vnet device is created for every VM that implies only one mac address per vnet device. The ebtables rules that I ran into enforce that. They prevent the VM from spoofing other mac addresses, for obvious security reasons. This lead to the filtering of all packages of my VPN clients, as their mac addresses differed from the VM's. That much for the reasons, now some additional words to the solution. Just flushing the ebtables table is not a persistent solution, as ovirt creates the rules every time a new vnet device is created. This happens on every reboot and every migration of the VM. Interesstingly, the filters are resistant to turning off the ebtables service with '# systemctl stop ebtables'. Although the service claims to be inactive, filtering takes place, nevertheless. I currently fail to find the website that pointed me to it, but the persistent solution is to disable the MAC anti spoofing filter. Here's how it goes: On the engine, do # engine-config -s EnableMACAntiSpoofingFilterRules=false # systemctl restart ovirt-engine After that, the ebtables rules are no longer applied to newly created vnets. (Filters for existing vnets are not removed, though) Maybe this helps someone, somewhen. Regards, Felix On 08/17/2015 02:05 PM, Felix Pepinghege wrote:
Hi again,
I just found the reason for the loss of the packages. Either ovirt or centos install some ebtables rules in the "nat" table. These rules filter ARP packages coming from vnet interfaces. Flushing the table solved my problems. However, I have no idea what the purpose of these rules is, so there might be unwanted side-effects to just flushing the rules away.. But for now I am happy!
Thanks to Ido for your help!
Regards, Felix
Am 17.08.2015 um 13:20 schrieb Felix Pepinghege:
Hi Ido, hi everybody,
sorry that I kept you waiting for two months, I only just found the time to go back to this problem.
You were completely right with your guess. The ethernet frames do appear on the vnet-interface, but not on the bridge. The dropped-counter seems to be independent from these losses, though.
However, while this tells me *where* the problem is, I still don't know *what* the problem is. I've done some research, but couldn't find anything particularly helpful. An interesting point may be that this problem is mono-directional. That is, the bridge happily passes vlan-tagged frames from the ethernet device to the vnet, but not the other way around. Untagged ethernet frames make their way through the brigde, no matter where they come from.
The vlan module is loaded, as to the versioning questions: # cat /etc/centos-release ; uname -s -v -r CentOS Linux release 7.1.1503 (Core) Linux 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015
The guest OS is an up-to-date Debian Jessie, which should not matter, though, as the frames get lost on the doorstep of the bridge on the host.
Again, any suggestions are much appreciated!
Regards, Felix
Am 16.06.2015 um 08:27 schrieb Ido Barkan:
Hey Felix.
IIUC your frames are dropped by the bridge. Ovirt uses Linux Bridges To connect virtual machines to 'networks'. The guest connects to the bridge using a tap device which usually is called 'vnet<number>'.
So, just to verify, can you please tcpdump both on the bridge device and on the tap device? The bridge can be quite noisy so I suggest filtering traffic using the guest's MAC address. So I am not sure what protocol you use for tunneling but applying a filter similar to this one should do the job: tcpdump -n -i vnet0 -vvv -s 1500 'udp[38:4]=0x001a4aaeec8e'
My guess is that you will observe traffic on the tap device, but not on the bridge. You didn't specify which centOS version you use but I do remember seeing people complaining about Linux bridges discarding their tagged frames. You can -maybe- also observe the 'dropped' counter increases on the bridge by running: 'ip -s link show dev trunk'
There were a few bugs on rhel6/7 about this, specifically I remember https://bugzilla.redhat.com/show_bug.cgi?id=1174291 and https://bugzilla.redhat.com/show_bug.cgi?id=1200275#c20
Also, is the vlan module loaded on your host? 'lsmod |grep 8021q'
Thanks, Ido
----- Original Message ----- From: "Felix Pepinghege" <pepinghege@ira.uka.de> To: Users@ovirt.org Sent: Monday, June 15, 2015 11:33:39 AM Subject: [ovirt-users] vlan-tagging on non-tagged network
Hi everybody!
I am experiencing a behaviour of ovirt, of which I don't know whether it is expected or not. My setup is as follows: A virtual machine has a logical network attached to it, which is configured without vlan-tagging and listens to the name 'trunk'. The VM is running an openvpn server. It is a patched openvpn version, including vlan-tagging. That is, openvpn clients get a vlan tag. This should not really be an issue but should satisfy the "why do you want to do it in the first place"-questions. Anyhow, effectively, the VM simply puts vlan-tagged ethernet-frames on the virtual network. These frames, however, never make it to the host's network bridge, which represents the logical network. My observations are: According to tcpdump, the vlan-tagged packages arrive at the "eth1"-interface inside the VM (which *is* the correct interface). Again, according to tcpdump, these packages never arrive at the corresponding network-bridge (i.e., the interface 'trunk') on the host. I know that the setup itself is feasible with KVM---I have it working on a proxmox-machine. Therefore, my conclusion is, that ovirt doesn't like vlan-tagged ethernet-frames on non-tagged logical networks, and somehow filters them out, though I don't really see on what "level" that would happen (Handling the ethernet frames should be a concern of KVM/QEMU/Linux only, once ovirt has started the VM). So this problem could be a CentOS issue, but I really don't see why CentOS should act differently than debian does (proxmox is debian-based). Is this a known/wanted/expected behaviour of ovirt, and can I somehow prevent or elude it?
Any help is much appreciated! Of course I am happy to provide more information if that helps helping me :)
Regards, Felix _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Tue, Aug 18, 2015 at 12:32:47PM +0200, Felix Pepinghege wrote:
Sorry for flooding the mailing list, but I have some new insights in why these things I described happen. So just in case someone stumbles over it in the future, I like to elaborate.
The problem is my specific use-case, that is, the VM being an openVPN server. By default, ovirt expects exactly one mac address per VM. As one vnet device is created for every VM that implies only one mac address per vnet device. The ebtables rules that I ran into enforce that. They prevent the VM from spoofing other mac addresses, for obvious security reasons. This lead to the filtering of all packages of my VPN clients, as their mac addresses differed from the VM's.
That much for the reasons, now some additional words to the solution. Just flushing the ebtables table is not a persistent solution, as ovirt creates the rules every time a new vnet device is created. This happens on every reboot and every migration of the VM. Interesstingly, the filters are resistant to turning off the ebtables service with '# systemctl stop ebtables'. Although the service claims to be inactive, filtering takes place, nevertheless. I currently fail to find the website that pointed me to it, but the persistent solution is to disable the MAC anti spoofing filter. Here's how it goes: On the engine, do # engine-config -s EnableMACAntiSpoofingFilterRules=false # systemctl restart ovirt-engine After that, the ebtables rules are no longer applied to newly created vnets. (Filters for existing vnets are not removed, though)
This disable mac spoofing protection for all VMs; even those which are less trust-worthy. A finer grained approach is to install vdsm-hook-macspoof engine-config -s "UserDefinedVMProperties=macspoof=(true|false);another_property=regexp" define a vnic profile with macspoof=true, and assign it to your VPN VM. http://www.ovirt.org/Vdsm_Hooks#Installing_a_hook
Hi Dan, On 08/19/2015 10:03 AM, Dan Kenigsberg wrote:
This disable mac spoofing protection for all VMs; even those which are less trust-worthy. A finer grained approach is to install vdsm-hook-macspoof
Thanks for pointing that out! I didn't know that this VM-specific solution exist. Regards, Felix
participants (3)
-
Dan Kenigsberg -
Felix Pepinghege -
Ido Barkan