We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets? -- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
Marcin / Lance, Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do this? We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400? Devin ---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org> We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets? -- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co -- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
Devin, oVirt does not currently support changing external network mtu from within ovirt (it rather relies on the provider handling this internally). If you are using OVN DHCP (have subnets defined for a network), you can modify the OVN DHCP options directly in the OVN database. I have never actually tested this myself, but looking at the OVN documentation, it should do the job on the ports. The standard OVN way to do so is to use the "ovn-vsctl set DHCP_Options ..." command. (Unfortunately as I am trying it now it tells me that modifying DHCP_Options is not supported) Alternatively, you can use the OVS python API (let me know if you need any help on this). Lance, Would changing the dhcp:options:mtu suffice? Could you please comment on how to modify the DHCP MTU using the OVN cmd line? Thanks, Marcin ----- Original Message -----
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 1:20:59 AM Subject: Fwd: oVirt / OVN / MTU
Marcin / Lance,
Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do this? We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400?
Devin
---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org>
We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets?
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
Hi Devin, This blog posting does a good job of explaining how to configure OVN DHCP support: http://blog.spinhirne.com/2016/09/an-introduction-to-ovn-routing.html The ovn-nb man page lists the DHCP options that can be provided, including mtu: http://openvswitch.org/support/dist-docs/ovn-nb.5.html And the ovn-nbctl man page has details about the command-line interface for setting DHCP options: http://openvswitch.org/support/dist-docs/ovn-nbctl.8.html I have very little experience using OVN's DHCP support, I've copied Numan in case I've left anything out. Lance ----- Original Message -----
From: "Marcin Mirecki" <mmirecki@redhat.com> To: "Devin Acosta" <devin@pabstatencio.com> Cc: "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 4:35:51 AM Subject: Re: oVirt / OVN / MTU
Devin,
oVirt does not currently support changing external network mtu from within ovirt (it rather relies on the provider handling this internally).
If you are using OVN DHCP (have subnets defined for a network), you can modify the OVN DHCP options directly in the OVN database. I have never actually tested this myself, but looking at the OVN documentation, it should do the job on the ports.
The standard OVN way to do so is to use the "ovn-vsctl set DHCP_Options ..." command. (Unfortunately as I am trying it now it tells me that modifying DHCP_Options is not supported) Alternatively, you can use the OVS python API (let me know if you need any help on this).
Lance, Would changing the dhcp:options:mtu suffice? Could you please comment on how to modify the DHCP MTU using the OVN cmd line?
Thanks, Marcin
----- Original Message -----
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 1:20:59 AM Subject: Fwd: oVirt / OVN / MTU
Marcin / Lance,
Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do this? We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400?
Devin
---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org>
We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets?
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
Hi Devin, Below is one example of creating dhcp options and associating them with logical ports using ovn-nbctl commands. Please see the links shared by Lance for more details. Lets say you have a network with cidr - 10.0.0.0/24. One example would be $ovn-nbctl dhcp-options-create 10.0.0.0/24 Run the command ovn-nbctl dhcp-options-list and store the uuid of it in any variable (DHCP_UUID) Now create the dhcp options for this DHCP_UUID just created. There are 4 dhcp options which needs to be defined (they are mandatory) - server_id - server_mac - router - lease_time since you want to add mtu option as well, you can add the dhcp options as $ovn-nbctl dhcp-options-set-options $DHCP_UUID server_id=10.0.0.1 server_mac=00:00:00:00:00:10 router=10.0.0.1 lease_time=3600 mtu=1400 The above is just an example. You can see the dhcp options set by running $ovn-nbctl dhcp-options-get-options $DHCP_UUID The dhcp options defined here have no value unless you associate these with the logical switch port. $ovn-nbctl lsp-set-dhcpv4-options $LPORT_NAME $DHCP_UUID Suppose if you have 3 logical ports - lp1, lp2 and lp3, you can associate it as $ovn-nbctl lsp-set-dhcpv4-options lp1 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp2 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp3 $DHCP_UUID Please let us know if you have any more questions. You can refer to this blog to get some more inner details of how native DHCP is supported in ovn - https://numansiddiqueblog.wordpress.com/2016/08/09/native-dhcp-support-in-ov... Thanks Numan On Mon, Dec 12, 2016 at 7:28 PM, Lance Richardson <lrichard@redhat.com> wrote:
Hi Devin,
This blog posting does a good job of explaining how to configure OVN DHCP support:
http://blog.spinhirne.com/2016/09/an-introduction-to-ovn-routing.html
The ovn-nb man page lists the DHCP options that can be provided, including mtu:
http://openvswitch.org/support/dist-docs/ ovn- nb.5.html <http://openvswitch.org/support/dist-docs/ovn-nb.5.html>
And the ovn-nbctl man page has details about the command-line interface for setting DHCP options:
http://openvswitch.org/support/dist-docs/ovn-nbctl.8.html
I have very little experience using OVN's DHCP support, I've copied Numan in case I've left anything out.
From: "Marcin Mirecki" <mmirecki@redhat.com> To: "Devin Acosta" <devin@pabstatencio.com> Cc: "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 4:35:51 AM Subject: Re: oVirt / OVN / MTU
Devin,
oVirt does not currently support changing external network mtu from within ovirt (it rather relies on the provider handling this internally).
If you are using OVN DHCP (have subnets defined for a network), you can modify the OVN DHCP options directly in the OVN database. I have never actually tested this myself, but looking at the OVN documentation, it should do the job on the ports.
The standard OVN way to do so is to use the "ovn-vsctl set DHCP_Options ..." command. (Unfortunately as I am trying it now it tells me that modifying DHCP_Options is not supported) Alternatively, you can use the OVS python API (let me know if you need any help on this).
Lance, Would changing the dhcp:options:mtu suffice? Could you please comment on how to modify the DHCP MTU using the OVN cmd line?
Thanks, Marcin
----- Original Message -----
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 1:20:59 AM Subject: Fwd: oVirt / OVN / MTU
Marcin / Lance,
Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do
Lance ----- Original Message ----- this?
We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400?
Devin
---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org>
We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets?
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
Marcin,Numan,Lance: I really appreciate all the assistance that you have given me thus far. I wanted to circle back on this topic, even though I sense I know what the answer will be. ;) My Networking team keeps insisting that they want to control DHCP from their side however yet still be able to create virtual Layer 2 networks within oVirt. I understand that it sounds like OVN was never meant for this kind of configuration. In the setup that we currently have going for the most part Linux boxes are working if we set the MTU to be lower around 1400, however with the Windows boxes we are getting very strange behavior, sometimes we have to set the MTU low to as 1000, but then the next day 1200 works. We aren't fully sure if it's just some strange windows issue with the driver or what. One question that was asked of me was if OVS/OVN supports like PMTUD, MTU Protocol discovery? Also I'm suspecting if was to configure oVIRT to use Neutron using OVS I would encounter the same issues I have now with the MTU if they are trying to do DHCP from a virtual appliance on the network? Any other comments or suggestions that you can provide on this? Thanks again. On Mon, Dec 12, 2016 at 7:14 AM, Numan Siddique <nusiddiq@redhat.com> wrote:
Hi Devin,
Below is one example of creating dhcp options and associating them with logical ports using ovn-nbctl commands. Please see the links shared by Lance for more details.
Lets say you have a network with cidr - 10.0.0.0/24.
One example would be
$ovn-nbctl dhcp-options-create 10.0.0.0/24
Run the command ovn-nbctl dhcp-options-list and store the uuid of it in any variable (DHCP_UUID) Now create the dhcp options for this DHCP_UUID just created. There are 4 dhcp options which needs to be defined (they are mandatory) - server_id - server_mac - router - lease_time
since you want to add mtu option as well, you can add the dhcp options as
$ovn-nbctl dhcp-options-set-options $DHCP_UUID server_id=10.0.0.1 server_mac=00:00:00:00:00:10 router=10.0.0.1 lease_time=3600 mtu=1400
The above is just an example. You can see the dhcp options set by running $ovn-nbctl dhcp-options-get-options $DHCP_UUID
The dhcp options defined here have no value unless you associate these with the logical switch port.
$ovn-nbctl lsp-set-dhcpv4-options $LPORT_NAME $DHCP_UUID
Suppose if you have 3 logical ports - lp1, lp2 and lp3, you can associate it as $ovn-nbctl lsp-set-dhcpv4-options lp1 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp2 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp3 $DHCP_UUID
Please let us know if you have any more questions.
You can refer to this blog to get some more inner details of how native DHCP is supported in ovn - https://numansiddiqueblog. wordpress.com/2016/08/09/native-dhcp-support-in-ovn/
Thanks Numan
On Mon, Dec 12, 2016 at 7:28 PM, Lance Richardson <lrichard@redhat.com> wrote:
Hi Devin,
This blog posting does a good job of explaining how to configure OVN DHCP support:
http://blog.spinhirne.com/2016/09/an-introduction-to-ovn-routing.html
The ovn-nb man page lists the DHCP options that can be provided, including mtu:
http://openvswitch.org/support/dist-docs/ ovn- nb.5.html <http://openvswitch.org/support/dist-docs/ovn-nb.5.html>
And the ovn-nbctl man page has details about the command-line interface for setting DHCP options:
http://openvswitch.org/support/dist-docs/ovn-nbctl.8.html
I have very little experience using OVN's DHCP support, I've copied Numan in case I've left anything out.
From: "Marcin Mirecki" <mmirecki@redhat.com> To: "Devin Acosta" <devin@pabstatencio.com> Cc: "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 4:35:51 AM Subject: Re: oVirt / OVN / MTU
Devin,
oVirt does not currently support changing external network mtu from within ovirt (it rather relies on the provider handling this internally).
If you are using OVN DHCP (have subnets defined for a network), you can modify the OVN DHCP options directly in the OVN database. I have never actually tested this myself, but looking at the OVN documentation, it should do the job on the ports.
The standard OVN way to do so is to use the "ovn-vsctl set DHCP_Options ..." command. (Unfortunately as I am trying it now it tells me that modifying DHCP_Options is not supported) Alternatively, you can use the OVS python API (let me know if you need any help on this).
Lance, Would changing the dhcp:options:mtu suffice? Could you please comment on how to modify the DHCP MTU using the OVN cmd line?
Thanks, Marcin
----- Original Message -----
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org , "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 1:20:59 AM Subject: Fwd: oVirt / OVN / MTU
Marcin / Lance,
Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do
Lance ----- Original Message ----- this?
We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400?
Devin
---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org>
We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets?
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
-- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
+Russel On Tue, Dec 20, 2016 at 10:30 AM, Devin Acosta <devin@pabstatencio.com> wrote:
Marcin,Numan,Lance:
I really appreciate all the assistance that you have given me thus far. I wanted to circle back on this topic, even though I sense I know what the answer will be. ;) My Networking team keeps insisting that they want to control DHCP from their side however yet still be able to create virtual Layer 2 networks within oVirt. I understand that it sounds like OVN was never meant for this kind of configuration.
It is not necessary to use native DHCP feature of OVN. You can always disable it. Infact it is disabled, unless dhcp options are added to each logical port. So it's fine to use your own DHCP server. I don't think MTU discovery is supported in OVN.
In the setup that we currently have going for the most part Linux boxes are working if we set the MTU to be lower around 1400, however with the Windows boxes we are getting very strange behavior, sometimes we have to set the MTU low to as 1000, but then the next day 1200 works. We aren't fully sure if it's just some strange windows issue with the driver or what.
One question that was asked of me was if OVS/OVN supports like PMTUD, MTU Protocol discovery? Also I'm suspecting if was to configure oVIRT to use Neutron using OVS I would encounter the same issues I have now with the MTU if they are trying to do DHCP from a virtual appliance on the network?
Any other comments or suggestions that you can provide on this?
Thanks again.
On Mon, Dec 12, 2016 at 7:14 AM, Numan Siddique <nusiddiq@redhat.com> wrote:
Hi Devin,
Below is one example of creating dhcp options and associating them with logical ports using ovn-nbctl commands. Please see the links shared by Lance for more details.
Lets say you have a network with cidr - 10.0.0.0/24.
One example would be
$ovn-nbctl dhcp-options-create 10.0.0.0/24
Run the command ovn-nbctl dhcp-options-list and store the uuid of it in any variable (DHCP_UUID) Now create the dhcp options for this DHCP_UUID just created. There are 4 dhcp options which needs to be defined (they are mandatory) - server_id - server_mac - router - lease_time
since you want to add mtu option as well, you can add the dhcp options as
$ovn-nbctl dhcp-options-set-options $DHCP_UUID server_id=10.0.0.1 server_mac=00:00:00:00:00:10 router=10.0.0.1 lease_time=3600 mtu=1400
The above is just an example. You can see the dhcp options set by running $ovn-nbctl dhcp-options-get-options $DHCP_UUID
The dhcp options defined here have no value unless you associate these with the logical switch port.
$ovn-nbctl lsp-set-dhcpv4-options $LPORT_NAME $DHCP_UUID
Suppose if you have 3 logical ports - lp1, lp2 and lp3, you can associate it as $ovn-nbctl lsp-set-dhcpv4-options lp1 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp2 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp3 $DHCP_UUID
Please let us know if you have any more questions.
You can refer to this blog to get some more inner details of how native DHCP is supported in ovn - https://numansiddiqueblog.wo rdpress.com/2016/08/09/native-dhcp-support-in-ovn/
Thanks Numan
On Mon, Dec 12, 2016 at 7:28 PM, Lance Richardson <lrichard@redhat.com> wrote:
Hi Devin,
This blog posting does a good job of explaining how to configure OVN DHCP support:
http://blog.spinhirne.com/2016/09/an-introduction-to-ovn-rou ting.html
The ovn-nb man page lists the DHCP options that can be provided, including mtu:
http://openvswitch.org/support/dist-docs/ ovn- nb.5.html <http://openvswitch.org/support/dist-docs/ovn-nb.5.html>
And the ovn-nbctl man page has details about the command-line interface for setting DHCP options:
http://openvswitch.org/support/dist-docs/ovn-nbctl.8.html
I have very little experience using OVN's DHCP support, I've copied Numan in case I've left anything out.
From: "Marcin Mirecki" <mmirecki@redhat.com> To: "Devin Acosta" <devin@pabstatencio.com> Cc: "users" <Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com
Sent: Monday, December 12, 2016 4:35:51 AM Subject: Re: oVirt / OVN / MTU
Devin,
oVirt does not currently support changing external network mtu from within ovirt (it rather relies on the provider handling this internally).
If you are using OVN DHCP (have subnets defined for a network), you can modify the OVN DHCP options directly in the OVN database. I have never actually tested this myself, but looking at the OVN documentation, it should do the job on the ports.
The standard OVN way to do so is to use the "ovn-vsctl set DHCP_Options ..." command. (Unfortunately as I am trying it now it tells me that modifying DHCP_Options is not supported) Alternatively, you can use the OVS python API (let me know if you need any help on this).
Lance, Would changing the dhcp:options:mtu suffice? Could you please comment on how to modify the DHCP MTU using the OVN cmd line?
Thanks, Marcin
----- Original Message -----
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org , "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 1:20:59 AM Subject: Fwd: oVirt / OVN / MTU
Marcin / Lance,
Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do
Lance ----- Original Message ----- this?
We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400?
Devin
---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org>
We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets?
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
One interesting thing I noticed today after looking at the logs from the 'ovn-controller' on the oVIRT nodes, is node1 is logging like crazy, the file is over 1.2GB in size already, seems to be looping like crazy? What should i check to see why it is logging and maxing CPU like this? 2016-12-20T21:16:17.517Z|91605|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.521Z|91606|poll_loop|INFO|Dropped 5666 log messages in last 6 seconds (most recently, 0 seconds ago) due to excessive rate 2016-12-20T21:16:17.521Z|91607|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (172.20.192.73:54710<->172.20.192.77:6642) at lib/stream-fd.c:155 (94% CPU usage) 2016-12-20T21:16:17.521Z|91608|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.524Z|91609|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.527Z|91610|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.531Z|91611|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.534Z|91612|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.538Z|91613|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.541Z|91614|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.545Z|91615|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.548Z|91616|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.552Z|91617|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.555Z|91618|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.559Z|91619|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.562Z|91620|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.566Z|91621|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.569Z|91622|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.573Z|91623|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.576Z|91624|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.580Z|91625|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.583Z|91626|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.587Z|91627|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.590Z|91628|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.593Z|91629|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.597Z|91630|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.600Z|91631|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.604Z|91632|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.607Z|91633|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.611Z|91634|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. top - 21:17:55 up 27 days, 3:49, 1 user, load average: 2.33, 2.12, 2.06 Tasks: 576 total, 2 running, 574 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.4 us, 0.3 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 13189636+total, 72442312 free, 41483504 used, 17970552 buff/cache KiB Swap: 4194300 total, 4193856 free, 444 used. 89843568 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21818 qemu 20 0 33.027g 0.031t 12176 S 94.1 25.5 465:24.64 qemu-kvm 29540 root 10 -10 51096 3844 1176 R 88.2 0.0 6:28.52 ovn-controller 1571 root 20 0 4376 672 520 S 29.4 0.0 6668:43 rngd 9282 qemu 20 0 1744736 703420 12108 S 5.9 0.5 35:34.51 qemu-kvm 30690 root 20 0 158228 2564 1496 R 5.9 0.0 0:00.02 top On Tue, Dec 20, 2016 at 1:47 AM, Numan Siddique <nusiddiq@redhat.com> wrote:
+Russel
On Tue, Dec 20, 2016 at 10:30 AM, Devin Acosta <devin@pabstatencio.com> wrote:
Marcin,Numan,Lance:
I really appreciate all the assistance that you have given me thus far. I wanted to circle back on this topic, even though I sense I know what the answer will be. ;) My Networking team keeps insisting that they want to control DHCP from their side however yet still be able to create virtual Layer 2 networks within oVirt. I understand that it sounds like OVN was never meant for this kind of configuration.
It is not necessary to use native DHCP feature of OVN. You can always disable it. Infact it is disabled, unless dhcp options are added to each logical port. So it's fine to use your own DHCP server. I don't think MTU discovery is supported in OVN.
In the setup that we currently have going for the most part Linux boxes are working if we set the MTU to be lower around 1400, however with the Windows boxes we are getting very strange behavior, sometimes we have to set the MTU low to as 1000, but then the next day 1200 works. We aren't fully sure if it's just some strange windows issue with the driver or what.
One question that was asked of me was if OVS/OVN supports like PMTUD, MTU Protocol discovery? Also I'm suspecting if was to configure oVIRT to use Neutron using OVS I would encounter the same issues I have now with the MTU if they are trying to do DHCP from a virtual appliance on the network?
Any other comments or suggestions that you can provide on this?
Thanks again.
On Mon, Dec 12, 2016 at 7:14 AM, Numan Siddique <nusiddiq@redhat.com> wrote:
Hi Devin,
Below is one example of creating dhcp options and associating them with logical ports using ovn-nbctl commands. Please see the links shared by Lance for more details.
Lets say you have a network with cidr - 10.0.0.0/24.
One example would be
$ovn-nbctl dhcp-options-create 10.0.0.0/24
Run the command ovn-nbctl dhcp-options-list and store the uuid of it in any variable (DHCP_UUID) Now create the dhcp options for this DHCP_UUID just created. There are 4 dhcp options which needs to be defined (they are mandatory) - server_id - server_mac - router - lease_time
since you want to add mtu option as well, you can add the dhcp options as
$ovn-nbctl dhcp-options-set-options $DHCP_UUID server_id=10.0.0.1 server_mac=00:00:00:00:00:10 router=10.0.0.1 lease_time=3600 mtu=1400
The above is just an example. You can see the dhcp options set by running $ovn-nbctl dhcp-options-get-options $DHCP_UUID
The dhcp options defined here have no value unless you associate these with the logical switch port.
$ovn-nbctl lsp-set-dhcpv4-options $LPORT_NAME $DHCP_UUID
Suppose if you have 3 logical ports - lp1, lp2 and lp3, you can associate it as $ovn-nbctl lsp-set-dhcpv4-options lp1 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp2 $DHCP_UUID $ovn-nbctl lsp-set-dhcpv4-options lp3 $DHCP_UUID
Please let us know if you have any more questions.
You can refer to this blog to get some more inner details of how native DHCP is supported in ovn - https://numansiddiqueblog.wo rdpress.com/2016/08/09/native-dhcp-support-in-ovn/
Thanks Numan
On Mon, Dec 12, 2016 at 7:28 PM, Lance Richardson <lrichard@redhat.com> wrote:
Hi Devin,
This blog posting does a good job of explaining how to configure OVN DHCP support:
http://blog.spinhirne.com/2016/09/an-introduction-to-ovn-rou ting.html
The ovn-nb man page lists the DHCP options that can be provided, including mtu:
http://openvswitch.org/support/dist-docs/ ovn- nb.5.html <http://openvswitch.org/support/dist-docs/ovn-nb.5.html>
And the ovn-nbctl man page has details about the command-line interface for setting DHCP options:
http://openvswitch.org/support/dist-docs/ovn-nbctl.8.html
I have very little experience using OVN's DHCP support, I've copied Numan in case I've left anything out.
Lance ----- Original Message -----
From: "Marcin Mirecki" <mmirecki@redhat.com> To: "Devin Acosta" <devin@pabstatencio.com> Cc: "users" <Users@ovirt.org>, "Lance Richardson" < lrichard@redhat.com> Sent: Monday, December 12, 2016 4:35:51 AM Subject: Re: oVirt / OVN / MTU
Devin,
oVirt does not currently support changing external network mtu from within ovirt (it rather relies on the provider handling this internally).
If you are using OVN DHCP (have subnets defined for a network), you can modify the OVN DHCP options directly in the OVN database. I have never actually tested this myself, but looking at the OVN documentation, it should do the job on the ports.
The standard OVN way to do so is to use the "ovn-vsctl set DHCP_Options ..." command. (Unfortunately as I am trying it now it tells me that modifying DHCP_Options is not supported) Alternatively, you can use the OVS python API (let me know if you need any help on this).
Lance, Would changing the dhcp:options:mtu suffice? Could you please comment on how to modify the DHCP MTU using the OVN cmd line?
Thanks, Marcin
----- Original Message -----
From: "Devin Acosta" <devin@pabstatencio.com> To: "Marcin Mirecki" <mmirecki@redhat.com>, "users" < Users@ovirt.org>, "Lance Richardson" <lrichard@redhat.com> Sent: Monday, December 12, 2016 1:20:59 AM Subject: Fwd: oVirt / OVN / MTU
Marcin / Lance,
Not sure if the list was working correctly, I couldn't see that my message below made it to the list. If I need to change the MTU settings for OVN / OpenVSwitch to something lower than 1500, what is the best way to do this? We noticed that some instances (ie: Windows 2012R2) are having issues with the default MTU of 1500, I think there is an issue at the upper layers, and we can get it to work if we manually set the MTU on the instance to say 1400. Is there an easy way to do this so that any VM's that come up automatically get MTU of 1400?
Devin
---------- Forwarded message ---------- From: Devin Acosta <devin@pabstatencio.com> Date: Fri, Dec 9, 2016 at 2:02 PM Subject: oVirt / OVN / MTU To: users <Users@ovirt.org>
We are running oVirt 4.0.5 and we have OVN working to provide a Virtual Layer 2 network. We are noticing that because the OVN is using Geneve and between all the firewalls and networks it crosses we are running into an MTU issue. What is the best suggested way to lower say the entire OVN network to say MTU of 1400, and also allow for fragmenting packets?
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
-- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
On Tue, Dec 20, 2016 at 4:18 PM, Devin Acosta <devin@pabstatencio.com> wrote:
One interesting thing I noticed today after looking at the logs from the 'ovn-controller' on the oVIRT nodes, is node1 is logging like crazy, the file is over 1.2GB in size already, seems to be looping like crazy? What should i check to see why it is logging and maxing CPU like this?
2016-12-20T21:16:17.517Z|91605|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.521Z|91606|poll_loop|INFO|Dropped 5666 log messages in last 6 seconds (most recently, 0 seconds ago) due to excessive rate 2016-12-20T21:16:17.521Z|91607|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (172.20.192.73:54710<->172.20.192.77:6642) at lib/stream-fd.c:155 (94% CPU usage) 2016-12-20T21:16:17.521Z|91608|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9.
Check "ovn-sbctl show" to figure out which host is chassis 252778b5-bc63-486d-8d82-c3e4350ee650. See if the other host has similar entries in its log. This could happen if a port is configured on two hosts at the same time. They will constantly fight over which chassis owns it. -- Russell Bryant
Chassis "c0dc0909-4274-400d-826e-82348fee47e9" hostname: "host1" Encap geneve ip: "172.20.192.73" options: {csum="true"} Port_Binding "f409d17f-38bc-4ee9-922b-63dbfc46cd91" Port_Binding "29dce6d5-77bd-4a3c-b077-bb002c4347d6" Port_Binding "e4f23f60-cc16-4399-a15b-fb05c09c4409" Port_Binding "a6d2008f-bb73-437b-bbac-b50487ea5257" Port_Binding "ee971db2-cdaf-4fa0-b83c-993371533af9" Port_Binding "f5aed0a0-03e0-46ec-92c6-07e5f03e9cc9" Port_Binding "4f1cf249-7fb6-4ff2-a7a9-a98bf45af6b8" Port_Binding "49725a3e-ef3f-474d-b84a-4975ce55e8fb" Port_Binding "bd7c66e9-81d5-4bfe-9331-203908048a6f" Port_Binding "301bf657-b6c4-459f-bd88-d21885190651" Port_Binding "0745689c-2c70-4b48-bda2-d0c1f8127133" Chassis "252778b5-bc63-486d-8d82-c3e4350ee650" hostname: "host2" Encap geneve ip: "172.20.192.75" options: {csum="true"} Port_Binding "d2e19c41-f8ff-49d2-bd8e-01d833e59a59" Chassis "45694909-7882-4d7a-948a-bdf34e6472cb" hostname: "host3" Encap geneve ip: "172.20.192.74" options: {csum="true"} Port_Binding "70443ae2-a04b-4d1f-90e4-f12e7e48fe82" Port_Binding "fcb6df47-32e6-4f7c-96fd-b598c33bff97" I see on host #3 2016-12-20T21:35:01.280Z|173706735|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.283Z|173706736|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.286Z|173706737|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.288Z|173706738|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.291Z|173706739|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.294Z|173706740|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.297Z|173706741|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.299Z|173706742|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.302Z|173706743|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.305Z|173706744|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.308Z|173706745|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.310Z|173706746|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.313Z|173706747|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.316Z|173706748|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.319Z|173706749|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.321Z|173706750|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.324Z|173706751|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.327Z|173706752|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. on host #, running ovs-vsctl show 61af799c-a621-445e-8183-23dcb38ea3cc Bridge br-int fail_mode: secure Port "vnet0" Interface "vnet0" error: "could not open network device vnet0 (No such device)" Port "ovn-c0dc09-0" Interface "ovn-c0dc09-0" type: geneve options: {csum="true", key=flow, remote_ip="172.20.192.73"} Port "ovn-456949-0" Interface "ovn-456949-0" type: geneve options: {csum="true", key=flow, remote_ip="172.20.192.74"} Port br-int Interface br-int type: internal ovs_version: "2.6.90" So some port got stuck on a box? Any idea what would cause this and what i should do to remove it? On Tue, Dec 20, 2016 at 2:29 PM, Russell Bryant <rbryant@redhat.com> wrote:
On Tue, Dec 20, 2016 at 4:18 PM, Devin Acosta <devin@pabstatencio.com> wrote:
One interesting thing I noticed today after looking at the logs from the 'ovn-controller' on the oVIRT nodes, is node1 is logging like crazy, the file is over 1.2GB in size already, seems to be looping like crazy? What should i check to see why it is logging and maxing CPU like this?
2016-12-20T21:16:17.517Z|91605|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.521Z|91606|poll_loop|INFO|Dropped 5666 log messages in last 6 seconds (most recently, 0 seconds ago) due to excessive rate 2016-12-20T21:16:17.521Z|91607|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (172.20.192.73:54710<->172.20.192.77:6642) at lib/stream-fd.c:155 (94% CPU usage) 2016-12-20T21:16:17.521Z|91608|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9.
Check "ovn-sbctl show" to figure out which host is chassis 252778b5-bc63-486d-8d82-c3e4350ee650. See if the other host has similar entries in its log.
This could happen if a port is configured on two hosts at the same time. They will constantly fight over which chassis owns it.
-- Russell Bryant
-- Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 || devin@linuxguru.co
I can't comment on how you may have gotten into this state, but to delete the port: ovs-vsctl del-port br-int vnet0 On Tue, Dec 20, 2016 at 4:38 PM, Devin Acosta <devin@pabstatencio.com> wrote:
Chassis "c0dc0909-4274-400d-826e-82348fee47e9" hostname: "host1" Encap geneve ip: "172.20.192.73" options: {csum="true"} Port_Binding "f409d17f-38bc-4ee9-922b-63dbfc46cd91" Port_Binding "29dce6d5-77bd-4a3c-b077-bb002c4347d6" Port_Binding "e4f23f60-cc16-4399-a15b-fb05c09c4409" Port_Binding "a6d2008f-bb73-437b-bbac-b50487ea5257" Port_Binding "ee971db2-cdaf-4fa0-b83c-993371533af9" Port_Binding "f5aed0a0-03e0-46ec-92c6-07e5f03e9cc9" Port_Binding "4f1cf249-7fb6-4ff2-a7a9-a98bf45af6b8" Port_Binding "49725a3e-ef3f-474d-b84a-4975ce55e8fb" Port_Binding "bd7c66e9-81d5-4bfe-9331-203908048a6f" Port_Binding "301bf657-b6c4-459f-bd88-d21885190651" Port_Binding "0745689c-2c70-4b48-bda2-d0c1f8127133" Chassis "252778b5-bc63-486d-8d82-c3e4350ee650" hostname: "host2" Encap geneve ip: "172.20.192.75" options: {csum="true"} Port_Binding "d2e19c41-f8ff-49d2-bd8e-01d833e59a59" Chassis "45694909-7882-4d7a-948a-bdf34e6472cb" hostname: "host3" Encap geneve ip: "172.20.192.74" options: {csum="true"} Port_Binding "70443ae2-a04b-4d1f-90e4-f12e7e48fe82" Port_Binding "fcb6df47-32e6-4f7c-96fd-b598c33bff97"
I see on host #3
2016-12-20T21:35:01.280Z|173706735|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.283Z|173706736|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.286Z|173706737|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.288Z|173706738|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.291Z|173706739|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.294Z|173706740|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.297Z|173706741|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.299Z|173706742|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.302Z|173706743|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.305Z|173706744|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.308Z|173706745|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.310Z|173706746|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.313Z|173706747|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.316Z|173706748|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.319Z|173706749|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.321Z|173706750|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.324Z|173706751|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650. 2016-12-20T21:35:01.327Z|173706752|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from c0dc0909-4274-400d-826e-82348fee47e9 to 252778b5-bc63-486d-8d82-c3e4350ee650.
on host #, running ovs-vsctl show
61af799c-a621-445e-8183-23dcb38ea3cc Bridge br-int fail_mode: secure Port "vnet0" Interface "vnet0" error: "could not open network device vnet0 (No such device)" Port "ovn-c0dc09-0" Interface "ovn-c0dc09-0" type: geneve options: {csum="true", key=flow, remote_ip="172.20.192.73"} Port "ovn-456949-0" Interface "ovn-456949-0" type: geneve options: {csum="true", key=flow, remote_ip="172.20.192.74"} Port br-int Interface br-int type: internal ovs_version: "2.6.90"
So some port got stuck on a box? Any idea what would cause this and what i should do to remove it?
On Tue, Dec 20, 2016 at 2:29 PM, Russell Bryant <rbryant@redhat.com> wrote:
On Tue, Dec 20, 2016 at 4:18 PM, Devin Acosta <devin@pabstatencio.com> wrote:
One interesting thing I noticed today after looking at the logs from the 'ovn-controller' on the oVIRT nodes, is node1 is logging like crazy, the file is over 1.2GB in size already, seems to be looping like crazy? What should i check to see why it is logging and maxing CPU like this?
2016-12-20T21:16:17.517Z|91605|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9. 2016-12-20T21:16:17.521Z|91606|poll_loop|INFO|Dropped 5666 log messages in last 6 seconds (most recently, 0 seconds ago) due to excessive rate 2016-12-20T21:16:17.521Z|91607|poll_loop|INFO|wakeup due to [POLLIN] on fd 12 (172.20.192.73:54710<->172.20.192.77:6642) at lib/stream-fd.c:155 (94% CPU usage) 2016-12-20T21:16:17.521Z|91608|binding|INFO|Changing chassis for lport d2e19c41-f8ff-49d2-bd8e-01d833e59a59 from 252778b5-bc63-486d-8d82-c3e4350ee650 to c0dc0909-4274-400d-826e-82348fee47e9.
Check "ovn-sbctl show" to figure out which host is chassis 252778b5-bc63-486d-8d82-c3e4350ee650. See if the other host has similar entries in its log.
This could happen if a port is configured on two hosts at the same time. They will constantly fight over which chassis owns it.
-- Russell Bryant
--
Devin Acosta Red Hat Certified Architect, LinuxStack 602-354-1220 <(602)%20354-1220> || devin@linuxguru.co
-- Russell Bryant
From: "Devin Acosta" <devin@pabstatencio.com> To: "Russell Bryant" <rbryant@redhat.com> Cc: "Numan Siddique" <nusiddiq@redhat.com>, "Lance Richardson" <lrichard@redhat.com>, "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org> Sent: Tuesday, December 20, 2016 4:38:32 PM Subject: Re: oVirt / OVN / MTU
So some port got stuck on a box? Any idea what would cause this and what i should do to remove it?
Maybe there's a better way, but you could run this on the two chassis that are involved: ovs-vsctl --format table --columns=name,external-ids list Interface And look for for port UUID from the log message, it should show up with "iface-id=" on both chassis (but should appear on only one). Once you figure out which one is wrong, do: ovs-vsctl remove Interface <port-name> external_ids iface-id=<interface-id>
From: "Lance Richardson" <lrichard@redhat.com> To: "Devin Acosta" <devin@pabstatencio.com> Cc: "Russell Bryant" <rbryant@redhat.com>, "Numan Siddique" <nusiddiq@redhat.com>, "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org> Sent: Tuesday, December 20, 2016 4:50:53 PM Subject: Re: oVirt / OVN / MTU
From: "Devin Acosta" <devin@pabstatencio.com> To: "Russell Bryant" <rbryant@redhat.com> Cc: "Numan Siddique" <nusiddiq@redhat.com>, "Lance Richardson" <lrichard@redhat.com>, "Marcin Mirecki" <mmirecki@redhat.com>, "users" <Users@ovirt.org> Sent: Tuesday, December 20, 2016 4:38:32 PM Subject: Re: oVirt / OVN / MTU
So some port got stuck on a box? Any idea what would cause this and what i should do to remove it?
Maybe there's a better way, but you could run this on the two chassis that are involved:
ovs-vsctl --format table --columns=name,external-ids list Interface
And look for for port UUID from the log message, it should show up with "iface-id=" on both chassis (but should appear on only one).
Once you figure out which one is wrong, do:
ovs-vsctl remove Interface <port-name> external_ids iface-id=<interface-id>
If you're running a recent version of ovs master, you might just need the fix listed below which was committed yesterday (maybe you don't have a duplicate port binding after all): commit f90bb0909c5320c2421cce392ad0d4ffaecb98e7 Author: Mickey Spiegel <mickeys.dev@gmail.com> Date: Tue Dec 20 13:23:46 2016 -0800 ovn-controller: Log chassis claiming lport only when changes occur. With recent OVN commits, the logic for a chassis to claim or release a logical port was consolidated. This is a good thing. However, there was a logic change that resulted in VLOG_INFO being generated every time on the ovn-controller. This patch changes the logic so that VLOG_INFO is only generated when there is a change, for example when the chassis claims an lport the first time. Signed-off-by: Mickey Spiegel <mickeys.dev@gmail.com> Signed-off-by: Ben Pfaff <blp@ovn.org>
On Tue, Dec 20, 2016 at 10:47 AM, Numan Siddique <nusiddiq@redhat.com> wrote:
+Russel
On Tue, Dec 20, 2016 at 10:30 AM, Devin Acosta <devin@pabstatencio.com> wrote:
Marcin,Numan,Lance:
I really appreciate all the assistance that you have given me thus far. I wanted to circle back on this topic, even though I sense I know what the answer will be. ;) My Networking team keeps insisting that they want to control DHCP from their side however yet still be able to create virtual Layer 2 networks within oVirt. I understand that it sounds like OVN was never meant for this kind of configuration.
It is not necessary to use native DHCP feature of OVN. You can always disable it. Infact it is disabled, unless dhcp options are added to each logical port. So it's fine to use your own DHCP server.
But note that in order for that to happen, you need your DHCP server to listen on all OVN overlays. You could do it by running it in an oVirt VM that is attached to all these overlays, or by adding per-overlay VM serving as a relay. Both idea seem cumbersome in opinion.
participants (6)
-
Dan Kenigsberg -
Devin Acosta -
Lance Richardson -
Marcin Mirecki -
Numan Siddique -
Russell Bryant