[Users] Setting up logical storage networks

[Trey Dockendorf]

Thanks for the response, see responses inline.

On Wed, Jun 20, 2012 at 2:54 AM, Mike Kolesnik <mkolesni at redhat.com> wrote:
> Hi,
>
> Please see reply in-line.
>
>> In ovirt-engine-3.1 I'm attempting to setup the base logical networks
>> and have run into 2 major issues.
>
> Are you using cluster version 3.0 or 3.1?
>

I have been using 3.1 as it's the default.  Is the different just the
API updates?  All I could really find related to 3.0 vs 3.1 pertaining
to networking was this document
http://www.ovirt.org/wiki/Features/Design/Network/SetupNetworks

>>
>> The first is I'm only seeing a Gateway field for the management
>> interface.  When I went to create a network for VMs (on seperate
>> subnet) I did not see a place to specify gateway (see img
>> ovirt_network_missing_gateway.png).  Right now my management port is
>> on a 100mbps network and the bridged devices live on a 1Gbps network
>> (net140 in cluster).  Is there a reason the gateway would be missing?
>> I've attached ovirt_networks.png that shows all the interfaces on my
>> host.
>
> Unfortunately, oVirt supports setting only the default gateway of the Host
> (This is the field you saw in the management network).
>
> We could theoretically use initscripts' static routing files, but that is left for
> future development.
>

So for now, is it then easier to just run all public interfaces
through the same subnet/gateway?  The main reason to run management
via 100Mbps and everything else 1Gbps was that our campus is out of
IPs so we're attempting to conserve on the usage of gigabit IPs.

>>
>> The second issue I'm having is creating a storage network.  I created
>> 2 logical networks , private0 and private1.  I left the "VM Network"
>> unchecked on both as my assumption was that dicates if they can be
>> added to VMs.  Since these are only for hosts to connect to the iSCSI
>> I didn't think that was necessary.  When I set the IP information
>> (private_network0.png) and select Ok the save goes through but when I
>> edit the interface again the information is gone and the file
>> ifcfg-eth4 does not have IP information.  This is what I looks like
>>
>> DEVICE=eth4
>> ONBOOT=yes
>> BOOTPROTO=none
>> HWADDR=00:1b:21:1d:33:f0
>> NM_CONTROLLED=no
>> MTU=9000
>
> I didn't quite understand what you did here..
> What I think you meant is:
> 1. You edited the network on a NIC, and provided static boot protocol
>   with the parameters (ip, netmask).
> 2. After that when you clicked OK then the configuration was sent to
>   the Host, and in the "Network Interfaces" tab for the Host you could
>   see the IP in the "Address" column. On the host the ifcfg script for
>   this network had these fields set.
> --- Assuming that no restart of Host or VDSM on Host was done ---
> 3. You edited the network again, didn't change anything, and clicked OK.
> 4. This time, the boot protocol info was gone from display & ifcfg file
>   on the Host.
>
> Is this correct?
>
> Also do you by any chance have the log files of ovirt (engine.log)/vdsm
> (vdsm.log) with the flow that you did?

I'll try to clarify the steps I took a little better, sorry if it was
unclear before.

1. Create logical network in Cluster that was NOT a "VM Network" (my
assumption of how to setup a storage network)
2. Edit NIC on host, set boot protocol to static and provide
IP/Netmask, and select the logical network created in #1, check "Save
network configuration"
3. After clicking OK the corresponding ifcfg file on the node was
modified, but the values for IP/Netmask were missing.  Also the values
did not appear in the network interface list, and were not shown when
going to that same interface and selecting "Add/Edit" again

That process did not involve a reboot of the host.

So in the host interface eth5 I set the following via web interface

Network: private1
Boot Protocol: Static
IP: 10.20.1.241
Subnet Mask: 255.0.0.0
Check: Save network configuration

After the save the node's ifcfg-eth5 is touched (based on modified
date in ls -la) but this is all it contains
DEVICE=eth5
ONBOOT=yes
BOOTPROTO=none
HWADDR=00:1b:21:1d:33:f1
NM_CONTROLLED=no
MTU=9000


As far as I can tell the only setting from ovirt-engine that made it
to that file was the MTU setting defined when creating the logical
network for the cluster.

Is my process somehow wrong or am I missing a step?  I've done this
with the node being in both "Up" status and "Maintenance", same
results.

As a test I manually updated the IP/Netmask of ifcfg-eth4 and it shows
up in the web interface with the correct information however any
changes via the web interface will remove the IPADDR and NETMASK
lines.

>
>>
>> I also attached image cluster_logical_networks.png that shows the all
>> the logical networks on this cluster.  So far my plan is to have a
>> single public interface for VM traffic, then two for storage traffic,
>> each going to a different switch.  This setup is just an initial test
>> but I'd hope to have it in production once I get some of these kinks
>> worked out.
>>
>> Please let me know what information would be useful to debug this
>> further.
>>
>> Thanks
>> - Trey
>
> Regards,
> Mike

Attached logs from the host and engine.  Host - node_vdsm.txt and
Engine - engine.txt.

The only issues I see are two deprecation notices from vdsm.

Thanks
- Trey
-------------- next part --------------
==> /var/log/ovirt-engine/engine.log <==
2012-06-20 10:29:42,834 INFO  [org.ovirt.engine.core.bll.AttachNetworkToVdsInterfaceCommand] (ajp--0.0.0.0-8009-2) [7b9ff58a] Running command: AttachNetworkToVdsInterfaceCommand internal: false. Entities affected :  ID: b19e1f70-ba2d-11e1-955f-525400053d53 Type: VDS
2012-06-20 10:29:42,841 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AddNetworkVDSCommand] (ajp--0.0.0.0-8009-2) [7b9ff58a] START, AddNetworkVDSCommand(vdsId = b19e1f70-ba2d-11e1-955f-525400053d53, networkName=private1, oldNetworkName=null, hostAddr=null, checkConnectivity=false, connectionTimeout=0, vlanId=null, bondName=null, nics=[eth5], inetAddr=10.20.1.241, networkMask=255.0.0.0, gateway=, stp=false, bondingOptions=null, bootProtocol=StaticIp, vmNetwork=false), log id: 4472c71e
2012-06-20 10:29:43,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AddNetworkVDSCommand] (ajp--0.0.0.0-8009-2) [7b9ff58a] FINISH, AddNetworkVDSCommand, log id: 4472c71e
2012-06-20 10:29:43,977 INFO  [org.ovirt.engine.core.bll.CommitNetworkChangesCommand] (ajp--0.0.0.0-8009-5) [163bbc82] Running command: CommitNetworkChangesCommand internal: false. Entities affected :  ID: b19e1f70-ba2d-11e1-955f-525400053d53 Type: VDS
2012-06-20 10:29:43,981 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (ajp--0.0.0.0-8009-5) [163bbc82] START, SetSafeNetworkConfigVDSCommand(vdsId = b19e1f70-ba2d-11e1-955f-525400053d53), log id: 2ba49f36
2012-06-20 10:29:44,034 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (ajp--0.0.0.0-8009-5) [163bbc82] FINISH, SetSafeNetworkConfigVDSCommand, log id: 2ba49f36
-------------- next part --------------
==> /var/log/vdsm/vdsm.log <==
Thread-41320::DEBUG::2012-06-20 10:26:32,349::BindingXMLRPC::872::vds::(wrapper) client [128.194.76.185]::call addNetwork with ('private1', '', '', ['eth5'], {'STP': 'no', 'NETMASK': '255.0.0.0', 'IPADDR': '10.20.1.241', 'bridged': 'false', 'mtu': '9000'}) {} flowID [7c97035b]
Thread-41320::WARNING::2012-06-20 10:26:32,349::API::1384::vds::(_translateOptionsToNew) options NETMASK is deprecated. Use netmask instead
Thread-41320::WARNING::2012-06-20 10:26:32,359::API::1384::vds::(_translateOptionsToNew) options IPADDR is deprecated. Use ipaddr instead
MainProcess|Thread-41320::DEBUG::2012-06-20 10:26:32,421::configNetwork::587::root::(addNetwork) validating bridge...
MainProcess|Thread-41320::INFO::2012-06-20 10:26:32,457::configNetwork::596::root::(addNetwork) Adding network private1 with vlan=None, bonding=None, nics=['eth5'], bondingOptions=None, mtu=9000, bridged=False, options={'STP': 'no'}
MainProcess|Thread-41320::DEBUG::2012-06-20 10:26:32,779::configNetwork::175::root::(_atomicBackup) Backed up /etc/sysconfig/network-scripts/ifcfg-eth5
MainProcess|Thread-41320::DEBUG::2012-06-20 10:26:32,782::configNetwork::221::root::(_persistentBackup) Persistently backed up /etc/sysconfig/network-scripts/ifcfg-eth5 (until next 'set safe config')
Thread-41320::DEBUG::2012-06-20 10:26:32,964::BindingXMLRPC::879::vds::(wrapper) return addNetwork with {'status': {'message': 'Done', 'code': 0}}
Thread-41322::DEBUG::2012-06-20 10:26:33,022::BindingXMLRPC::872::vds::(wrapper) client [128.194.76.185]::call getCapabilities with () {} flowID [7c97035b]

==> /var/log/vdsm/libvirt.log <==
2012-06-20 15:26:33.023+0000: 25190: debug : virConnectNumOfNetworks:8369 : conn=0xcdcdc0
2012-06-20 15:26:33.025+0000: 25190: debug : virConnectListNetworks:8407 : conn=0xcdcdc0, names=0x7f181006ff10, maxnames=4
2012-06-20 15:26:33.026+0000: 25190: debug : virNetworkLookupByName:8531 : conn=0xcdcdc0, name=vdsm-ovirtmgmt
2012-06-20 15:26:33.028+0000: 25190: debug : virNetworkGetXMLDesc:9036 : network=0x7f181019d410, flags=0
2012-06-20 15:26:33.030+0000: 25190: debug : virNetworkLookupByName:8531 : conn=0xcdcdc0, name=vdsm-net140
2012-06-20 15:26:33.032+0000: 25190: debug : virNetworkFree:8875 : network=0x7f181019d410
2012-06-20 15:26:33.032+0000: 25190: debug : virNetworkGetXMLDesc:9036 : network=0x7f1810163f90, flags=0
2012-06-20 15:26:33.033+0000: 25190: debug : virNetworkLookupByName:8531 : conn=0xcdcdc0, name=vdsm-private0
2012-06-20 15:26:33.035+0000: 25190: debug : virNetworkFree:8875 : network=0x7f1810163f90
2012-06-20 15:26:33.035+0000: 25190: debug : virNetworkGetXMLDesc:9036 : network=0x7f1810150790, flags=0
2012-06-20 15:26:33.037+0000: 25190: debug : virNetworkLookupByName:8531 : conn=0xcdcdc0, name=vdsm-private1
2012-06-20 15:26:33.038+0000: 25190: debug : virNetworkFree:8875 : network=0x7f1810150790
2012-06-20 15:26:33.038+0000: 25190: debug : virNetworkGetXMLDesc:9036 : network=0x7f18100b62d0, flags=0
2012-06-20 15:26:33.040+0000: 25190: debug : virNetworkFree:8875 : network=0x7f18100b62d0

==> /var/log/vdsm/vdsm.log <==
Thread-41322::DEBUG::2012-06-20 10:26:33,061::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm' (cwd None)
Thread-41322::DEBUG::2012-06-20 10:26:33,096::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Thread-41322::DEBUG::2012-06-20 10:26:33,097::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img' (cwd None)
Thread-41322::DEBUG::2012-06-20 10:26:33,132::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Thread-41322::DEBUG::2012-06-20 10:26:33,133::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None)
Thread-41322::DEBUG::2012-06-20 10:26:33,163::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Thread-41322::DEBUG::2012-06-20 10:26:33,164::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None)
Thread-41322::DEBUG::2012-06-20 10:26:33,195::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Thread-41322::DEBUG::2012-06-20 10:26:33,195::__init__::1164::Storage.Misc.excCmd::(_log) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None)
Thread-41322::DEBUG::2012-06-20 10:26:33,227::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Thread-41322::DEBUG::2012-06-20 10:26:33,229::BindingXMLRPC::879::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:3ceae4f3f21c'}], 'FC': []}, 'packages2': {'kernel': {'release': '220.17.1.el6.x86_64', 'buildtime': 1337144497.0, 'version': '2.6.32'}, 'spice-server': {'release': '5.el6', 'buildtime': '1323304307', 'version': '0.8.2'}, 'vdsm': {'release': '0.33.git40b2d55.el6', 'buildtime': '1339678794', 'version': '4.10.0'}, 'qemu-kvm': {'release': '2.209.el6_2.5', 'buildtime': '1336984850', 'version': '0.12.1.2'}, 'libvirt': {'release': '23.el6_2.9', 'buildtime': '1339580410', 'version': '0.9.4'}, 'qemu-img': {'release': '2.209.el6_2.5', 'buildtime': '1336984850', 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(Fake) CPU', 'hooks': {'before_vm_start': {'10_simpleqemu': {'md5': '2c88a35172c02f2125fafe39c1c95fa9'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'addr': '165.91.83.87', 'cfg': {'UUID': '5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03', 'IPADDR': '165.91.83.87', 'DNS1': '128.194.254.1', 'DEFROUTE': 'yes', 'NETMASK': '255.255.255.0', 'GATEWAY': '165.91.83.1', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'PREFIX': '24', 'BOOTPROTO': 'none', 'DEVICE': 'ovirtmgmt', 'IPV4_FAILURE_FATAL': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes', 'IPV6INIT': 'no'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '165.91.83.1', 'ports': ['eth0']}, 'net140': {'addr': '165.91.140.37', 'cfg': {'IPADDR': '165.91.140.37', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.192', 'STP': 'no', 'DEVICE': 'net140', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.192', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['eth1']}}, 'uuid': '00000000-0000-0000-0000-00144F8E43E3_00:14:4f:8e:06:2e', 'lastClientIface': 'ovirtmgmt', 'nics': {'eth7': {'hwaddr': '00:14:4f:8e:06:31', 'netmask': '', 'speed': 0, 'addr': '', 'mtu': '1500'}, 'eth6': {'hwaddr': '00:14:4f:8e:06:30', 'netmask': '', 'speed': 0, 'addr': '', 'mtu': '1500'}, 'eth5': {'hwaddr': '00:1b:21:1d:33:f1', 'netmask': '', 'speed': 0, 'addr': '', 'mtu': '9000'}, 'eth4': {'hwaddr': '00:1b:21:1d:33:f0', 'netmask': '255.0.0.0', 'speed': 1000, 'addr': '10.20.1.240', 'mtu': '9000'}, 'eth1': {'hwaddr': '00:14:4f:8e:06:2f', 'netmask': '', 'speed': 1000, 'addr': '', 'mtu': '1500'}, 'eth0': {'hwaddr': '00:14:4f:8e:06:2e', 'netmask': '', 'speed': 100, 'addr': '', 'mtu': '1500'}}, 'software_revision': '0.33', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': 'pge,clflush,syscall,vme,tsc,vmx,cmov,nx,rep_good,pat,lm,msr,fpu,fxsr,pae,cmp_legacy,fxsr_opt,mmx,cx8,extd_apicid,mce,de,mca,pse,3dnowext,ht,apic,sse,3dnow,pni,rdtscp,svm,mmxext,sse2,cr8_legacy,lahf_lm,cx16,extapic,pse36,mtrrmodel_486,model_pentium,model_pentium2,model_pentium3,model_pentiumpro,model_qemu32,model_coreduo,model_core2duo,model_n270,model_Conroe,model_Penryn,model_Nehalem,model_Opteron_G1', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:3ceae4f3f21c', 'netConfigDirty': 'True', 'memSize': '7870', 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'cpuSpeed': '2393.373', 'cpuSockets': '2', 'vlans': {}, 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0', '3.1'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.2.0', u'pc', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '2.el6.centos.7', 'version': '6', 'name': 'oVirt Node'}, 'lastClient': '128.194.76.185'}}
Thread-41323::DEBUG::2012-06-20 10:26:34,061::BindingXMLRPC::872::vds::(wrapper) client [128.194.76.185]::call setSafeNetworkConfig with () {} flowID [c076fc0]
Thread-41323::DEBUG::2012-06-20 10:26:34,070::BindingXMLRPC::879::vds::(wrapper) return setSafeNetworkConfig with {'status': {'message': 'Done', 'code': 0}}
Thread-41325::DEBUG::2012-06-20 10:26:36,782::task::588::TaskManager.Task::(_updateState) Task=`6700a880-4927-463e-aa42-7b4ff8b1fba4`::moving from state init -> state preparing
Thread-41325::INFO::2012-06-20 10:26:36,783::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-41325::INFO::2012-06-20 10:26:36,783::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-41325::DEBUG::2012-06-20 10:26:36,783::task::1172::TaskManager.Task::(prepare) Task=`6700a880-4927-463e-aa42-7b4ff8b1fba4`::finished: {}
Thread-41325::DEBUG::2012-06-20 10:26:36,783::task::588::TaskManager.Task::(_updateState) Task=`6700a880-4927-463e-aa42-7b4ff8b1fba4`::moving from state preparing -> state finished
Thread-41325::DEBUG::2012-06-20 10:26:36,784::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-41325::DEBUG::2012-06-20 10:26:36,784::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-41325::DEBUG::2012-06-20 10:26:36,784::task::978::TaskManager.Task::(_decref) Task=`6700a880-4927-463e-aa42-7b4ff8b1fba4`::ref 0 aborting False
Thread-41331::DEBUG::2012-06-20 10:26:47,680::task::588::TaskManager.Task::(_updateState) Task=`fa4af826-4aea-47d4-b7cd-8a81c81c4ef7`::moving from state init -> state preparing
Thread-41331::INFO::2012-06-20 10:26:47,680::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-41331::INFO::2012-06-20 10:26:47,680::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-41331::DEBUG::2012-06-20 10:26:47,681::task::1172::TaskManager.Task::(prepare) Task=`fa4af826-4aea-47d4-b7cd-8a81c81c4ef7`::finished: {}
Thread-41331::DEBUG::2012-06-20 10:26:47,681::task::588::TaskManager.Task::(_updateState) Task=`fa4af826-4aea-47d4-b7cd-8a81c81c4ef7`::moving from state preparing -> state finished
Thread-41331::DEBUG::2012-06-20 10:26:47,681::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-41331::DEBUG::2012-06-20 10:26:47,681::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-41331::DEBUG::2012-06-20 10:26:47,681::task::978::TaskManager.Task::(_decref) Task=`fa4af826-4aea-47d4-b7cd-8a81c81c4ef7`::ref 0 aborting False