[Users] Unable to finish AIO 3.3.0 - VDSM

Tue Sep 24 16:48:13 UTC 2013

> Date: Tue, 24 Sep 2013 13:28:26 +0100
> From: danken at redhat.com
> To: cybertimber2000 at hotmail.com
> CC: jbrooks at redhat.com; masayag at redhat.com; alonbl at redhat.com; users at ovirt.org
> Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM
> 
> On Mon, Sep 23, 2013 at 06:10:09PM -0400, Nicholas Kesick wrote:
> > Ok the thread got a little fragmented, so I'm trying to merge these
> > together. Let me know if I missed something.
> >  
> > ----- Original Message -----
> > > > From: "Dan Kenigsberg" <danken at redhat.com>
> > > > To: "Jason Brooks" <jbrooks at redhat.com>
> > > > Cc: "Nicholas Kesick" <cybertimber2000 at hotmail.com>, "oVirt Mailing List" <users at ovirt.org>
> > > > Sent: Monday, September 23, 2013 1:23:28 PM
> > > > Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM
> > > > 
> > > > On Mon, Sep 23, 2013 at 03:29:10PM -0400, Jason Brooks wrote:
> > > > > 
> > > > > 
> > > > > > 
> > > > > > Hi Nicholas, I just installed an F19 AIO without any problem. My install
> > > > > > was
> > > > > > only minimal, though. I restored my snapshot to pre-ovirt install and
> > > > > > added
> > > > > > the "standard" group, rebooted, installed, and vdsm still installed
> > > > > > normally.
> > > > > > 
> > > > > > I'm wondering if it makes a difference if the system starts out with
> > > > > > minimal+standard, rather than starting out minimal and adding standard
> > > > > > after...
> > > > > > 
> > > > > > This is with dhcp addressing.
> > > > > 
> > > > > Another difference -- my AIO machine has nics w/ the regular eth0 naming --
> > > > > don't know if the biosdevname bits could be causing an issue...
> > > > 
> > > > Would I be wrong to assume that you had
> > > > /etc/sysconfig/network-scripts/ifcfg-eth0 defined before installation
> > > > began?
> > > 
> > > My systems do always have this defined before installation begins. I almost always
> > > do PXE installs of Fedora. Wonder how it differs from a DVD install...
> > > 
> > > Jason
> > Good question. My particular attempts with ovirt 3.3 have been by
> > using the netinstall.iso. I can try a DVD install with
> > minimal+standard. For what it's worth that's what I've always used,
> > especially after that thread about minimal missing tar, and that part
> > of the install or setup requires tar. 
> >  
> > I do wonder if the interface names are messing things up. I don't know
> > if something changed upstream, or if it's part of the net install, but
> > interfaces aren't name eth# or em# (embedded) / p#p# (PCI) anymore.
> > Mine are way more cryptic now (enp4s0) and it's very annoying.
> > I know there wasn't a ifcfg-eth0, but there is a ifcfg-enp4s0.
> > ifconfig currently reports that I'm using em1, but there is no config
> > file for that. hmm.
> 
> Naming per se should not matter. I have seen ovirt install on hosts with
> all kinds of nic names.
> 
> However could we get to the bottom of the relation between enp4s0 and
> em1? Do you have two physical nics, or just one? Which of them is
> physically connected to the outer world? Your /var/log/messages that
> it's your em1. THAT nic should have it ifcfg file before Vdsm is
> installed on the host.
> 
There is only one NIC on the system, an NIC that is embedded to the motherboard.
During install it's listed as enp4s0. Not sure what it's called after first or second boot, but there is a ifcfg-enp4s0 for it.
Currently on the system, the output of ifconfig doesn't list that interface, but instead lists em1. If I try a ifdown enp4s0, em1 goes down. It's like they are linked, but I can't find any reference of that. It's off at the moment so when I can boot it up I'll provide more info.
I might reinstall and see how it progresses from being named enp4s0 to em1. Worst case I'll disable biosdevname.
> >  
> > > On Mon, Sep 23, 2013 at 02:25:49PM -0400, Moti Asayag wrote:
> > > > I have looked at the getVdsCapabilities reported by VDSM for the first time, on which the engine based its
> > > > setupNetwork command for configuring the management network:
> > > > 
> > > > 'lastClientIface': 'em1',
> > > > 'nics': {'em1': {'netmask': '255.255.255.0', 'addr': '192.168.2.9', 'hwaddr': 'a4:ba:db:ec:ea:cd', 'cfg': {}, 'ipv6addrs': > ['fe80::a6ba:dbff:feec:eacd/64', '2001:4830:1692:1:a6ba:dbff:feec:eacd/64'], 'speed': 1000, 'mtu': '1500'}}
> > > > 
> > > > Based on that input, the engine sends setupNetwork command to configure the management network on top of 'em1' nic.
> > > > However, since it has no bootprotocol or gateway, it is identified as bootproto=NONE, which result in engine not to pass ip > address/subnet/gateway to vdsm, therefore the command fails.
> > > 
> > > This seems very similar to what triggered
> > > 
> > >  Bug 987813 - [RFE] report BOOTPROTO and BONDING_OPTS independent of
> > >  netdevice.cfg
> > > 
> > > Vdsm does not really cope with network definitions that are not
> > > ifcfg-based. I do not know what makes Fedora 19 sometimes use ifcfg
> > > out-of-the-box and sometimes not, but in order to play well with the
> > > current implementation of Vdsm, we should make it use legacy initscript
> > > for network definition.
> > > 
> > See the above. I wonder if netinstall.iso messes with this.
> > > > 
> > > > networks=[ovirtmgmt {id=b05f272d-473f-41cb-b5a0-25cf06951f6b, description=Management Network, comment=null, subnet=null, gateway=null, > type=null, vlanId=null, stp=false, dataCenterId=7a7e4a7a-ecc8-4b3e-b200-2b54d0809a52, mtu=0, vmNetwork=true, cluster=NetworkCluster {id=> {clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true, migration=false}, providedBy=null, label=null}],
> > > > bonds=[],
> > > > interfaces=[em1 {id=80a1e7c8-05bd-4209-a6fe-a23c38538330, vdsId=5e3da5d9-7133-4119-86c0-f655d3e37b38, name=em1, > macAddress=a4:ba:db:ec:ea:cd, networkName=ovirtmgmt, bondName=null, bootProtocol=NONE, address=192.168.2.9, subnet=255.255.255.0, > gateway=null, mtu=0, bridged=true, speed=1000, type=2, networkImplementationDetails=null}],
> > > > removedNetworks=[],
> > > > 
> > > > As received by VDSM, where we can see the ip address and the subnet are omitted:
> > > > Thread-17::DEBUG::2013-09-23 07:07:21,314::BindingXMLRPC::979::vds::(wrapper) client [192.168.2.9]::call setupNetworks with > ({'ovirtmgmt': {'nic': 'em1', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
> > > > 
> > > > Looking at the traceback on vdsm.log reveals less information:
> > > > storageRefresh::DEBUG::2013-09-23 07:07:22,597::misc::817::SamplingMethod::(__call__) Returning last result
> > > > Thread-17::ERROR::2013-09-23 07:09:24,154::API::1261::vds::(setupNetworks)
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/API.py", line 1259, in setupNetworks
> > > > supervdsm.getProxy().setupNetworks(networks, bondings, options)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in setupNetworks
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
> > > > raise convert_to_error(kind, result)
> > > > ConfigNetworkError: (29, '')
> > > > 
> > > > 
> > > > Nicholas, could you add also /var/log/vdsm/supervdsm.log and /var/log/messages so we can get more input about
> > > > the failure to bring up em1 ?
> > Yep, attached.
> >  		 	   		  
> 
<snipped>
> 
> Here, Vdsm is trying to configure em1 with no ip address (because it
> found no ifcfg-em1 to begin with). But then, it fails to do so since
> NetworkManager is still running.
> 
> So if possible, make sure ifcfg-em1 exists (and has the correct
> BOOTPROT=dhcp in it) and the NetworkManager is off before initiating
> installation. That's annoying, I know. It should be fix, for sure. But
> currently it is a must.
> 
> Regards,
> Dan.
Hopefully I didn't miss any other comments in that log snippet of log file ^^;; It's good to know why it keeps failing. I'm just trying to figure out how to move forward from here, and I'll take a crack at it this evening.
I thought that NetworkManager only needs to be disabled if you are using a static IP? I did try disabling NM before I realized it said only for static and had a failure, but probably because of the interface/ifcfg issue. I will try again this evening.

I'll try to jump into IRC by 5pm EDT if you happen to be around.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20130924/34429812/attachment-0001.html>