Re: [ovirt-users] oVirt 3.5 & NAT

Monday, 17 November 2014

Finally got chance to test this but unfortunately it did not work.  I did make a change to
replace:

 HOOK_NAME = 'extnet_pg'

with

 HOOK_NAME = 'extnet_ovs'

and on the engine it looks okay:

CustomDeviceProperties: {type=interface;prop={extnet_ovs=^[a-zA-Z0-9_ ---]+(:[a-zA-Z0-9_
---]+|)$}} version: 3.5

and when I checked vdsm.log it is being called okay when the VM first starts after adding
the additional NICs.

How am I able to debug please ?

Thanks, Phil

----- Original Message -----
From: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
To: "Phil Daws" <uxbod(a)splatnix.net&gt;
Cc: users(a)ovirt.org
Sent: Sunday, 2 November, 2014 2:21:20 PM
Subject: Re: [ovirt-users] oVirt 3.5 & NAT

----- Original Message -----
...
 From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 To: users(a)ovirt.org
 Sent: Saturday, November 1, 2014 9:31:53 AM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 Is there anybody who can help with this please ? Of course
...
  Am pretty sure all I need
 to be able to do know is create a network hook that changes a VM XML from:

       <interface type='bridge'>
         <mac address='00:1a:4a:83:fb:00'/>
         <source bridge='ovirtmgmt'/>
         <target dev='vnet0'/>
         <model type='virtio'/>
         <filterref filter='vdsm-no-mac-spoofing'/>
         <link state='up'/>
         <bandwidth>
         </bandwidth>
         <alias name='net0'/>
         <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
         function='0x0'/>
       </interface>

 to something like:

       <interface type='network'>
         <mac address='00:1a:4a:83:fb:00'/>
         <source network='ovs-network' portgroup='vlan-20'/>
         <model type='virtio'/>
         <link state='up'/>
         <alias name='net0'/>
         <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
         function='0x0'/>
       </interface>

 so replacing the bridge with a network and setting the network to be the OVS
 one I have defined.

 Any ideas please ? Thanks, Phil 
If you just need to do that it's relatively simple. You just need to create a
python executable and put it in:
-  /usr/libexec/vdsm/hooks/before_device_create
-  /usr/libexec/vdsm/hooks/before_nic_hotplug

Then, the content should be something like the following:

    #!/usr/bin/env python
    """
    Tweak an interface defintion so that it uses a specific network and port group.
    It applies on a per vnic basis, it gets triggered and used by two different
    events:
        * before_device_create
        * before_nic_hotplug

    This hook can be used to force a VM to use a libvirt network that is managed
    outside of ovirt, such as an openvswitch network, or libvirt's default network.
    """

    import os
    import sys
    import traceback
    import xml.dom

    import hooking

    HOOK_NAME = 'extnet_pg'

    def replaceSource(interface, newnet, port_group=None):
        source, = interface.getElementsByTagName('source')
        source.removeAttribute('bridge')
        source.setAttribute('network', newnet)
        if port_group is not None:
            source.setAttribute('portgroup', port_group)
        interface.setAttribute('type', 'network')

    def main():
        newnet = os.environ.get('extnet')
        if ':' in newnet:
            newnet, port_group = newnet.split(':')
        if newnet is not None:
            doc = hooking.read_domxml()
            interface, = doc.getElementsByTagName('interface')
            replaceSource(interface, newnet)
            hooking.write_domxml(doc)

    def test():

        interface = xml.dom.minidom.parseString("""
        <interface type="bridge">
            <address bus="0x00" domain="0x0000"
function="0x0" slot="0x03"\
                                                type="pci"/>
            <mac address="00:1a:4a:16:01:b0"/>
            <model type="virtio"/>
            <source bridge="ovirtmgmt"/>
            <filterref filter="vdsm-no-mac-spoofing"/>
            <link state="up"/>
            <boot order="1"/>
        </interface>
        """).getElementsByTagName('interface')[0]

        print "Interface before forcing network: %s" % \
            interface.toxml(encoding='UTF-8')

        replaceSource(interface, 'yipee')
        print "Interface after forcing network: %s" % \
            interface.toxml(encoding='UTF-8')

        replaceSource(interface, 'ovs-network', port_group='vlan-20')
        print "Interface after forcing network and port group: %s" % \
            interface.toxml(encoding='UTF-8')

    if __name__ == '__main__':
        try:
            if '--test' in sys.argv:
                test()
            else:
                main()
        except:
            hooking.exit_hook('extnet hook: [unexpected error]: %s\n' %
                              traceback.format_exc())

This is a modification of the existing extnet hook. Then, to use it, go to the
engine machine's command line and do:

    sudo engine-config -s CustomDeviceProperties=\
    '{type=interface;prop={extnet_pg=^[a-zA-Z0-9_ ---]+(:[a-zA-Z0-9_ ---]+|)$}}'

Verify that it was properly added
    sudo engine-config -g CustomDeviceProperties

Restart the engine.

On the webadmin:

    Define a vNIC profile for a network that has a 'extnet_pg' custom device
    property with value 'ovs-network:vlan-20', for example.

    Then, attach the defined profile to the relevant vNIC. When the VM is run,
    the relevant vNIC will be attached to the network and portgroup that you
    passed it.

I haven't tested it, but it should work or need very minimal modification.

Let me know how it goes ;-)

Regards,

Toni
...

 ----- Original Message -----
 From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 To: users(a)ovirt.org
 Sent: Tuesday, 28 October, 2014 12:18:47 PM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 Am starting to believe may have to give up on oVirt and move back to straight
 KVM and Openvswitch as that just worked which is a real shame :(

 ----- Original Message -----
 From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 To: users(a)ovirt.org
 Sent: Monday, 27 October, 2014 5:51:53 PM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 I believe am really missing the point somewhere along the line :( on my
 straight KVM and OVS system I see:

 brctl show
 bridge name	bridge id		STP enabled	interfaces
 virbr0		8000.fe5400552ed9	yes		vnet5
 							vnet8
 ovs-vsctl show
 55a2af2f-daf5-4f01-a757-9bccaf4f6932
     Bridge "ovsbr0"
         Port "vnet0"
             Interface "vnet0"
         Port "vnet1"
             tag: 8
             Interface "vnet1"
         Port "vnet13"
             tag: 14
             Interface "vnet13"
         Port "vnet9"
             tag: 10
             Interface "vnet9"
         Port "mgmt0"
             Interface "mgmt0"
                 type: internal
         Port "vnet14"
             tag: 8
             Interface "vnet14"
         Port "ovsbr0"
             Interface "ovsbr0"
                 type: internal
         Port "vnet11"
             tag: 8
             Interface "vnet11"

 and on the new oVirt system I see:

 brctl show
 bridge name	bridge id		STP enabled	interfaces
 ;vdsmdummy;		8000.000000000000	no
 ovirtmgmt		8000.c81f66c42c76	no		em1
 							vnet0
 ovs-vsctl show
 08554d11-3ba7-4303-b9d5-6a09f23c9057
     Bridge "ovsbr0"
         Port "ovsbr0"
             Interface "ovsbr0"
                 type: internal

 so I have the ovirtmgmt bridge running fine and that is responding to
 traffic.  If I bond an IP to the ovsbr0 it responds okay:

 ip add add 88.XXX.XXX.XXX/29 dev ovsbr0

 so network traffic is flowing in and out of OVS.  The next step would be to
 create a VM that will act as the firewall with a public facing interface on
 ovsbr0, but then all internal networks would need specific vlans behind
 that.  I have already defined a new network called ovs-network:

 <network>
   <name>ovs-network</name>
   <uuid>76152e72-34da-43bc-864a-85e727025bc9</uuid>
   <forward mode='bridge'/>
   <bridge name='ovsbr0' />
   <virtualport type='openvswitch'/>
   <portgroup name='vlan-08'>
     <vlan>
       <tag id='8'/>
     </vlan>
   </portgroup>
   <portgroup name='vlan-10'>
     <vlan>
       <tag id='10'/>
     </vlan>
   </portgroup>
   <portgroup name='vlan-13'>
     <vlan>
       <tag id='13'/>
     </vlan>
   </portgroup>
   <portgroup name='vlan-14'>
     <vlan>
       <tag id='14'/>
     </vlan>
   </portgroup>
   <portgroup name='vlan-20'>
     <vlan>
       <tag id='20'/>
     </vlan>
   </portgroup>
   <portgroup name='vlan-99' default='yes'>
   </portgroup>
 </network>

 and then I should just be able to assign the network and vlan IDs to each VM
 NIC:

     <interface type='network'>
       <mac address='54:52:00:02:01:02'/>
       <source network='ovs-network' portgroup='vlan-08'/>
       <model type='virtio'/>
       <address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
       function='0x0'/>
     </interface>
     <interface type='network'>
       <mac address='54:52:00:02:01:03'/>
       <source network='ovs-network' portgroup='vlan-10'/>
       <model type='virtio'/>
       <address type='pci' domain='0x0000' bus='0x00'
slot='0x09'
       function='0x0'/>
     </interface>

 so where should I start to put all this together please ? I believe am 90%
 there its just how to get the source-network and port-group to be assigned
 to a VM guest when its built ?  Guess probably need to create a new hook
 similar to
 https://github.com/oVirt/vdsm/blob/master/vdsm_hooks/macspoof/macspoof_vn...
 that changes this in each guests VM definition from bridge to a network:

       <interface type='bridge'>
         <mac address='00:1a:4a:83:fb:00'/>
         <source bridge='ovirtmgmt'/>
         <target dev='vnet0'/>
         <model type='virtio'/>
         <filterref filter='vdsm-no-mac-spoofing'/>
         <link state='up'/>
         <bandwidth>
         </bandwidth>
         <alias name='net0'/>
         <address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
         function='0x0'/>
       </interface>

 Thanks, Phil
 ----- Original Message -----
 From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 To: users(a)ovirt.org
 Sent: Monday, 27 October, 2014 5:16:05 PM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 How does one add multiple custom properties ? tried:

 engine-config -s
 CustomDeviceProperties='{type=interface;prop={vlan=^[a-zA-Z0-9_
 ---]+$}}{type=interface;prop={bridge=^[a-zA-Z0-9_ ---]+$}}'

 but ended up with one call vlan and the other prop :) If can add vlan and
 bridge then should be able to use a vNIC profile for adding an interface
 directly to OVS using a custom hook.

 Thanks, Phil

 ----- Original Message -----
 From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 To: users(a)ovirt.org
 Sent: Monday, 27 October, 2014 3:04:20 PM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 Well, in fact have got something to work now! Left ovirtmgmt and em1 alone
 but ran:

 $ ovs-vsctl add-br ovsbr0
 $ ip link add name veth0 type veth peer name veth1
 $ brctl addif ovirtmgmt veth0
 $ ovs-vsctl add-port ovsbr veth1
 $ ip add add XXX.XXX.XXX.XXX/29 dev veth1
 $ ip link set veth0 up && ip link set veth1 up

 and now veth1 is responding as-well as veth0.

 ovs-vsctl show
 08554d11-3ba7-4303-b9d5-6a09f23c9057
 Bridge "ovsbr0"
 Port "veth1"
 Interface "veth1"
 Port "ovsbr0"
 Interface "ovsbr0"
 type: internal

 so what I think should do now is create a custom parameter on the Engine
 Manager that allows one to define an OVS bridge name and VLAN so when a
 virtual guest is created it can be assigned to the new bridge; with the use
 of a custom hook.

 Thanks, Phil
 ----- Original Message -----
 From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 To: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
 Cc: users(a)ovirt.org
 Sent: Monday, 27 October, 2014 2:10:34 PM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 Darn, looks like this will not work :( the problem is that oVirt creates the
 bridge ovirtmgmt and binds that to your interface eg. em1. So at that point
 you have network running. If you then try to add that to the OVS stack your
 networking stop :( I tried to add it as a port using ovs-vsctl add-port
 ovsbr0 ovirtmgmt which is accepted but then networking stops. As soon as I
 remove again networking comes back to life. There does not seem to be a way
 to have two co-existing bridges :( Thanks, Phil

 ----- Original Message -----
 From: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
 To: "Phil Daws" <uxbod(a)splatnix.net&gt;
 Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, users(a)ovirt.org
 Sent: Monday, 27 October, 2014 12:13:30 PM
 Subject: Re: [ovirt-users] oVirt 3.5 & NAT

 ----- Original Message -----
 > From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 > To: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
 > Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, users(a)ovirt.org
 > Sent: Monday, October 27, 2014 11:41:56 AM
 > Subject: Re: [ovirt-users] oVirt 3.5 & NAT
 > 
 > Hi Antoni:
 > 
 > Yes, prior to the reboot it did work okay. This is how it should look I
 > believe:
 > 
 > Bridge "ovirtmgmt"
 > Port "mgmt0"
 > Interface "mgmt0"
 > type: internal
 > Port "ovsbr0"
 > Interface "ovsbr0"
 > type: internal
 > 
 > So the bridge would be defined by oVirt then I guess with a custom hook
 > that
 > would then be added to the OVS stack ?

 exactly! You could just make a hook script that runs an after_network_setup
 hook that does the ovs-vsctl for you ;-)

 Here you can see the presentation I gave last February at devconf about
 extending
 with configurators and hooks.

 http://blog.antoni.me/devconf14/#/8/1

 I linked directly to a before_network_setup hook sample, because it works
 just like
 the after_network_setup hook. Instead of logging to systemd, just add that if
 'remove' is not in data and network == 'ovirtmgmt', it adds the network
 bridge to
 the vswitch with python's subprocess.call or subprocess.check_output.

 You can send it if you want me to take a look ;-)

 PS: It is possible to write the hooks in bash, c, perl, etc. But we only have
 the
 convenience read_json methods and such for python. If you wanted to, you
 could have
 a simple bash hook that just checked if there was an ovirtmgmt bridge and it
 would
 add it doing ovs-vsctl in the before_vdsm_start hooking point. That would
 have the
 drawback that changing the ovirtmgmt bridge with oVirt UI would leave it
 disconnected
 again.

 > 
 > Thanks, Phil
 > 
 > ----- Original Message -----
 > From: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
 > To: "Phil Daws" <uxbod(a)splatnix.net&gt;
 > Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, users(a)ovirt.org
 > Sent: Monday, 27 October, 2014 9:56:38 AM
 > Subject: Re: [ovirt-users] oVirt 3.5 & NAT
 > 
 > 
 > 
 > ----- Original Message -----
 > > From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 > > To: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
 > > Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, users(a)ovirt.org
 > > Sent: Monday, October 27, 2014 10:37:18 AM
 > > Subject: Re: [ovirt-users] oVirt 3.5 & NAT
 > > 
 > > That is what I tried but oVirt appears to overwrite the bridge
 > > information
 > > on
 > > boot :( Thanks, Phil
 > 
 > But before rebooting, does it work as you intended? If so, you could just
 > make
 > a vdsm hook that adds ovirtmgmt to the ovs bridge after it is set up. (I
 > could
 > give more directions into how to do it).
 > 
 > > 
 > > ----- Original Message -----
 > > From: "Antoni Segura Puimedon" <asegurap(a)redhat.com&gt;
 > > To: "Phil Daws" <uxbod(a)splatnix.net&gt;
 > > Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, users(a)ovirt.org
 > > Sent: Monday, 27 October, 2014 8:00:33 AM
 > > Subject: Re: [ovirt-users] oVirt 3.5 & NAT
 > > 
 > > 
 > > 
 > > ----- Original Message -----
 > > > From: "Phil Daws" <uxbod(a)splatnix.net&gt;
 > > > To: "Dan Kenigsberg" <danken(a)redhat.com&gt;
 > > > Cc: users(a)ovirt.org
 > > > Sent: Saturday, October 25, 2014 5:02:59 PM
 > > > Subject: Re: [ovirt-users] oVirt 3.5 & NAT
 > > > 
 > > > Hmmm, this is becoming difficult ..
 > > > 
 > > > I have added into the engine the custom hook and understand how that
 > > > will
 > > > work. The issue is how can a single NIC use two different bridges ?
 > > > Example with OVS would be that one requires:
 > > > 
 > > > em1 -+ ovirtmgmt (bridge) -> management IP (public)
 > > > + ovs (bridge) -> firewall IP (public)
 > > > | 
 > > > + vlan 1
 > > > + vlan 2
 > > > 
 > > > this works fine when using OVS and KVM, without oVirt, so there must be
 > > > a
 > > > way
 > > > to hook the two together without a Neutron appliance.
 > > > 
 > > > Any thoughts ? Thanks, Phil.
 > > 
 > > I haven't tried this, and it may not work, but what happens if you add
 > > the
 > > ovirtmgmt
 > > bridge as a port of the ovs bridge?
 > > > 
 > > > 
 > > > ----- Original Message -----
 > > > From: "Dan Kenigsberg" <danken(a)redhat.com&gt;
 > > > To: "Phil Daws" <uxbod(a)splatnix.net&gt;
 > > > Cc: users(a)ovirt.org
 > > > Sent: Wednesday, 22 October, 2014 3:54:46 PM
 > > > Subject: Re: [ovirt-users] oVirt 3.5 & NAT
 > > > 
 > > > On Wed, Oct 22, 2014 at 03:12:09PM +0100, Phil Daws wrote:
 > > > > Thanks Dan & Antoni:
 > > > > 
 > > > > I wonder then if I could replace the standard libvirt defined
network
 > > > > with
 > > > > an OpenVSwitch one like I have on my dev system? That is just
 > > > > straight
 > > > > KVM with OVS integrated. Maybe a bit more overhead in administration
 > > > > but
 > > > > possibly less than having to spin up a Neutron Appliance.
 > > > 
 > > > Once you start to use the vdsm-hook-extnet, all that you need to do is
 > > > to replace the libvirt-side definition of the "external
network". This
 > > > may well be an OpenVSwitch-based network e.g.
 > > > http://libvirt.org/formatnetwork.html#elementVlanTag
 > > > _______________________________________________
 > > > Users mailing list
 > > > Users(a)ovirt.org
 > > > http://lists.ovirt.org/mailman/listinfo/users
 > > > 
 > > 
 > 
 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
 _______________________________________________
 Users mailing list
 Users(a)ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] oVirt 3.5 & NAT