Ok the thread got a little fragmented=2C so I'm trying to merge these together. Let me know if I missed something. =20 ----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Jason Brooks" <jbrooks@redhat.com> Cc: "Nicholas Kesick" <cybertimber2000@hotmail.com>=2C "oVirt Maili= ng List" <users@ovirt.org> Sent: Monday=2C September 23=2C 2013 1:23:28 PM Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM =20 On Mon=2C Sep 23=2C 2013 at 03:29:10PM -0400=2C Jason Brooks wrote:
=20 =20
=20 Hi Nicholas=2C I just installed an F19 AIO without any problem.= My install was only minimal=2C though. I restored my snapshot to pre-ovirt ins= tall and added the "standard" group=2C rebooted=2C installed=2C and vdsm still= installed normally. =20 I'm wondering if it makes a difference if the system starts out= with minimal+standard=2C rather than starting out minimal and adding= standard after... =20 This is with dhcp addressing. =20 Another difference -- my AIO machine has nics w/ the regular eth0= naming -- don't know if the biosdevname bits could be causing an issue... =20 Would I be wrong to assume that you had /etc/sysconfig/network-scripts/ifcfg-eth0 defined before installati= on began? =20 My systems do always have this defined before installation begins. I = almost always do PXE installs of Fedora. Wonder how it differs from a DVD install..= . =20 Jason Good question. My particular attempts with ovirt 3.3 have been by using the netinstall.iso. I can try a DVD install with minimal+standard. For what it's worth that's what I've always used=2C especially after that thread about minimal missing tar=2C and that part of the install or setup requires tar.=20 =20 I do wonder if the interface names are messing things up. I don't know if something changed upstream=2C or if it's part of the net install=2C = but interfaces aren't name eth# or em# (embedded) / p#p# (PCI) anymore. Mine are way more cryptic now (enp4s0) and it's very annoying. I know there wasn't a ifcfg-eth0=2C but there is a ifcfg-enp4s0. ifconfig currently reports that I'm using em1=2C but there is no config file for that. hmm. =20 Naming per se should not matter. I have seen ovirt install on hosts with all kinds of nic names. =20 However could we get to the bottom of the relation between enp4s0 and em1? Do you have two physical nics=2C or just one? Which of them is
Date: Tue=2C 24 Sep 2013 13:28:26 +0100 From: danken@redhat.com To: cybertimber2000@hotmail.com CC: jbrooks@redhat.com=3B masayag@redhat.com=3B alonbl@redhat.com=3B user= s@ovirt.org Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM =20 On Mon=2C Sep 23=2C 2013 at 06:10:09PM -0400=2C Nicholas Kesick wrote: physically connected to the outer world? Your /var/log/messages that it's your em1. THAT nic should have it ifcfg file before Vdsm is installed on the host. =20 There is only one NIC on the system=2C an NIC that is embedded to the mothe= rboard. During install it's listed as enp4s0. Not sure what it's called after first= or second boot=2C but there is a ifcfg-enp4s0 for it. Currently on the system=2C the output of ifconfig doesn't list that interfa= ce=2C but instead lists em1. If I try a ifdown enp4s0=2C em1 goes down. It'= s like they are linked=2C but I can't find any reference of that. It's off = at the moment so when I can boot it up I'll provide more info. I might reinstall and see how it progresses from being named enp4s0 to em1.= Worst case I'll disable biosdevname.
=20
On Mon=2C Sep 23=2C 2013 at 02:25:49PM -0400=2C Moti Asayag wrote:
I have looked at the getVdsCapabilities reported by VDSM for the fi= rst time=2C on which the engine based its setupNetwork command for configuring the management network: =20 'lastClientIface': 'em1'=2C 'nics': {'em1': {'netmask': '255.255.255.0'=2C 'addr': '192.168.2.9= '=2C 'hwaddr': 'a4:ba:db:ec:ea:cd'=2C 'cfg': {}=2C 'ipv6addrs': > ['fe80::a= 6ba:dbff:feec:eacd/64'=2C '2001:4830:1692:1:a6ba:dbff:feec:eacd/64']=2C 'sp= eed': 1000=2C 'mtu': '1500'}} =20 Based on that input=2C the engine sends setupNetwork command to con=
--_9b96a2b2-e7ae-4c34-91de-2f1f6da28755_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable =0A= =0A= =0A= figure the management network on top of 'em1' nic.
However=2C since it has no bootprotocol or gateway=2C it is identif= ied as bootproto=3DNONE=2C which result in engine not to pass ip > address/= subnet/gateway to vdsm=2C therefore the command fails. =20 This seems very similar to what triggered =20 Bug 987813 - [RFE] report BOOTPROTO and BONDING_OPTS independent of netdevice.cfg =20 Vdsm does not really cope with network definitions that are not ifcfg-based. I do not know what makes Fedora 19 sometimes use ifcfg out-of-the-box and sometimes not=2C but in order to play well with th= e current implementation of Vdsm=2C we should make it use legacy initsc= ript for network definition. =20 See the above. I wonder if netinstall.iso messes with this. =20 networks=3D[ovirtmgmt {id=3Db05f272d-473f-41cb-b5a0-25cf06951f6b=2C= description=3DManagement Network=2C comment=3Dnull=2C subnet=3Dnull=2C gat= eway=3Dnull=2C > type=3Dnull=2C vlanId=3Dnull=2C stp=3Dfalse=2C dataCenterI= d=3D7a7e4a7a-ecc8-4b3e-b200-2b54d0809a52=2C mtu=3D0=2C vmNetwork=3Dtrue=2C = cluster=3DNetworkCluster {id=3D> {clusterId=3Dnull=2C networkId=3Dnull}=2C = status=3DOPERATIONAL=2C display=3Dfalse=2C required=3Dtrue=2C migration=3Df= alse}=2C providedBy=3Dnull=2C label=3Dnull}]=2C bonds=3D[]=2C interfaces=3D[em1 {id=3D80a1e7c8-05bd-4209-a6fe-a23c38538330=2C vds= Id=3D5e3da5d9-7133-4119-86c0-f655d3e37b38=2C name=3Dem1=2C > macAddress=3Da= 4:ba:db:ec:ea:cd=2C networkName=3Dovirtmgmt=2C bondName=3Dnull=2C bootProto=
col=3DNONE=2C address=3D192.168.2.9=2C subnet=3D255.255.255.0=2C > gateway= =3Dnull=2C mtu=3D0=2C bridged=3Dtrue=2C speed=3D1000=2C type=3D2=2C network= ImplementationDetails=3Dnull}]=2C > > > > removedNetworks=3D[]=2C > > > >=20 > > > > As received by VDSM=2C where we can see the ip address and the subn= et are omitted: > > > > Thread-17::DEBUG::2013-09-23 07:07:21=2C314::BindingXMLRPC::979::vd= s::(wrapper) client [192.168.2.9]::call setupNetworks with > ({'ovirtmgmt':= {'nic': 'em1'=2C 'STP': 'no'=2C 'bridged': 'true'}}=2C {}=2C {'connectivit= yCheck': 'true'=2C 'connectivityTimeout': 120}) {} > > > >=20 > > > > Looking at the traceback on vdsm.log reveals less information: > > > > storageRefresh::DEBUG::2013-09-23 07:07:22=2C597::misc::817::Sampli= ngMethod::(__call__) Returning last result > > > > Thread-17::ERROR::2013-09-23 07:09:24=2C154::API::1261::vds::(setup= Networks) > > > > Traceback (most recent call last): > > > > File "/usr/share/vdsm/API.py"=2C line 1259=2C in setupNetworks > > > > supervdsm.getProxy().setupNetworks(networks=2C bondings=2C options) > > > > File "/usr/share/vdsm/supervdsm.py"=2C line 50=2C in __call__ > > > > return callMethod() > > > > File "/usr/share/vdsm/supervdsm.py"=2C line 48=2C in <lambda> > > > > **kwargs) > > > > File "<string>"=2C line 2=2C in setupNetworks > > > > File "/usr/lib64/python2.7/multiprocessing/managers.py"=2C line 773= =2C in _callmethod > > > > raise convert_to_error(kind=2C result) > > > > ConfigNetworkError: (29=2C '') > > > >=20 > > > >=20 > > > > Nicholas=2C could you add also /var/log/vdsm/supervdsm.log and /var= /log/messages so we can get more input about > > > > the failure to bring up em1 ? > > Yep=2C attached. > > =20 >=20 <snipped> >=20 > Here=2C Vdsm is trying to configure em1 with no ip address (because it > found no ifcfg-em1 to begin with). But then=2C it fails to do so since > NetworkManager is still running. >=20 > So if possible=2C make sure ifcfg-em1 exists (and has the correct > BOOTPROT=3Ddhcp in it) and the NetworkManager is off before initiating > installation. That's annoying=2C I know. It should be fix=2C for sure. Bu= t > currently it is a must. >=20 > Regards=2C > Dan. Hopefully I didn't miss any other comments in that log snippet of log file = ^^=3B=3B It's good to know why it keeps failing. I'm just trying to figure = out how to move forward from here=2C and I'll take a crack at it this eveni= ng. I thought that NetworkManager only needs to be disabled if you are using a = static IP? I did try disabling NM before I realized it said only for static= and had a failure=2C but probably because of the interface/ifcfg issue. I = will try again this evening.
>=3B >=3B >=3B >=3B >=3B >=3B minimal+standard=2C rather than = starting out minimal and adding standard<br>>=3B >=3B >=3B >=3B >= =3B >=3B after...<br>>=3B >=3B >=3B >=3B >=3B >=3B <br>>=3B= >=3B >=3B >=3B >=3B >=3B This is with dhcp addressing.<br>>=3B= >=3B >=3B >=3B >=3B <br>>=3B >=3B >=3B >=3B >=3B Another= difference -- my AIO machine has nics w/ the regular eth0 naming --<br>>= =3B >=3B >=3B >=3B >=3B don't know if the biosdevname bits could be= causing an issue...<br>>=3B >=3B >=3B >=3B <br>>=3B >=3B >= =3B >=3B Would I be wrong to assume that you had<br>>=3B >=3B >=3B = >=3B /etc/sysconfig/network-scripts/ifcfg-eth0 defined before installatio= n<br>>=3B >=3B >=3B >=3B began?<br>>=3B >=3B >=3B <br>>=3B = >=3B >=3B My systems do always have this defined before installation be= gins. I almost always<br>>=3B >=3B >=3B do PXE installs of Fedora. Wo= nder how it differs from a DVD install...<br>>=3B >=3B >=3B <br>>= =3B >=3B >=3B Jason<br>>=3B >=3B Good question. My particular attem=
I'll try to jump into IRC by 5pm EDT if you happen to be around. =0A= = --_9b96a2b2-e7ae-4c34-91de-2f1f6da28755_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> </head> <body class=3D'hmmessage'><div dir=3D'ltr'>=0A= =0A= <style><!--=0A= .hmmessage P=0A= {=0A= margin:0px=3B=0A= padding:0px=0A= }=0A= body.hmmessage=0A= {=0A= font-size: 12pt=3B=0A= font-family:Calibri=0A= }=0A= --></style>=0A= <div dir=3D"ltr"><br><br><div>>=3B Date: Tue=2C 24 Sep 2013 13:28:26 +010= 0<br>>=3B From: danken@redhat.com<br>>=3B To: cybertimber2000@hotmail.c= om<br>>=3B CC: jbrooks@redhat.com=3B masayag@redhat.com=3B alonbl@redhat.= com=3B users@ovirt.org<br>>=3B Subject: Re: [Users] Unable to finish AIO = 3.3.0 - VDSM<br>>=3B <br>>=3B On Mon=2C Sep 23=2C 2013 at 06:10:09PM -0= 400=2C Nicholas Kesick wrote:<br>>=3B >=3B Ok the thread got a little f= ragmented=2C so I'm trying to merge these<br>>=3B >=3B together. Let me= know if I missed something.<br>>=3B >=3B <br>>=3B >=3B ----- Orig= inal Message -----<br>>=3B >=3B >=3B >=3B From: "Dan Kenigsberg" &l= t=3Bdanken@redhat.com>=3B<br>>=3B >=3B >=3B >=3B To: "Jason Brook= s" <=3Bjbrooks@redhat.com>=3B<br>>=3B >=3B >=3B >=3B Cc: "Nicho= las Kesick" <=3Bcybertimber2000@hotmail.com>=3B=2C "oVirt Mailing List"= <=3Busers@ovirt.org>=3B<br>>=3B >=3B >=3B >=3B Sent: Monday=2C= September 23=2C 2013 1:23:28 PM<br>>=3B >=3B >=3B >=3B Subject: Re= : [Users] Unable to finish AIO 3.3.0 - VDSM<br>>=3B >=3B >=3B >=3B = <br>>=3B >=3B >=3B >=3B On Mon=2C Sep 23=2C 2013 at 03:29:10PM -040= 0=2C Jason Brooks wrote:<br>>=3B >=3B >=3B >=3B >=3B <br>>=3B &= gt=3B >=3B >=3B >=3B <br>>=3B >=3B >=3B >=3B >=3B >=3B <b= r>>=3B >=3B >=3B >=3B >=3B >=3B Hi Nicholas=2C I just installed= an F19 AIO without any problem. My install<br>>=3B >=3B >=3B >=3B = >=3B >=3B was<br>>=3B >=3B >=3B >=3B >=3B >=3B only minimal= =2C though. I restored my snapshot to pre-ovirt install and<br>>=3B >= =3B >=3B >=3B >=3B >=3B added<br>>=3B >=3B >=3B >=3B >=3B= >=3B the "standard" group=2C rebooted=2C installed=2C and vdsm still ins= talled<br>>=3B >=3B >=3B >=3B >=3B >=3B normally.<br>>=3B >= =3B >=3B >=3B >=3B >=3B <br>>=3B >=3B >=3B >=3B >=3B >= =3B I'm wondering if it makes a difference if the system starts out with<br= pts with ovirt 3.3 have been by<br>>=3B >=3B using the netinstall.iso. = I can try a DVD install with<br>>=3B >=3B minimal+standard. For what it= 's worth that's what I've always used=2C<br>>=3B >=3B especially after = that thread about minimal missing tar=2C and that part<br>>=3B >=3B of = the install or setup requires tar. <br>>=3B >=3B <br>>=3B >=3B I d= o wonder if the interface names are messing things up. I don't know<br>>= =3B >=3B if something changed upstream=2C or if it's part of the net inst= all=2C but<br>>=3B >=3B interfaces aren't name eth# or em# (embedded) /= p#p# (PCI) anymore.<br>>=3B >=3B Mine are way more cryptic now (enp4s0= ) and it's very annoying.<br>>=3B >=3B I know there wasn't a ifcfg-eth0= =2C but there is a ifcfg-enp4s0.<br>>=3B >=3B ifconfig currently report= s that I'm using em1=2C but there is no config<br>>=3B >=3B file for th= at. hmm.<br>>=3B <br>>=3B Naming per se should not matter. I have seen = ovirt install on hosts with<br>>=3B all kinds of nic names.<br>>=3B <br=
>=3B However could we get to the bottom of the relation between enp4s0 a= nd<br>>=3B em1? Do you have two physical nics=2C or just one? Which of th= em is<br>>=3B physically connected to the outer world? Your /var/log/mess= ages that<br>>=3B it's your em1. THAT nic should have it ifcfg file befor= e Vdsm is<br>>=3B installed on the host.<br>>=3B <br>There is only one = NIC on the system=2C an NIC that is embedded to the motherboard.<br>During = install it's listed as enp4s0. Not sure what it's called after first or sec= ond boot=2C but there is a ifcfg-enp4s0 for it.<br>Currently on the system= =2C the output of ifconfig doesn't list that interface=2C but instead lists= em1. If I try a ifdown enp4s0=2C em1 goes down. It's like they are linked= =2C but I can't find any reference of that. It's off at the moment so when = I can boot it up I'll provide more info.<br>I might reinstall and see how i= t progresses from being named enp4s0 to em1. Worst case I'll disable biosde= vname.<br>>=3B >=3B <br>>=3B >=3B >=3B On Mon=2C Sep 23=2C 2013 = at 02:25:49PM -0400=2C Moti Asayag wrote:<br>>=3B >=3B >=3B >=3B I = have looked at the getVdsCapabilities reported by VDSM for the first time= =2C on which the engine based its<br>>=3B >=3B >=3B >=3B setupNetwo= rk command for configuring the management network:<br>>=3B >=3B >=3B = >=3B <br>>=3B >=3B >=3B >=3B 'lastClientIface': 'em1'=2C<br>>= =3B >=3B >=3B >=3B 'nics': {'em1': {'netmask': '255.255.255.0'=2C 'ad= dr': '192.168.2.9'=2C 'hwaddr': 'a4:ba:db:ec:ea:cd'=2C 'cfg': {}=2C 'ipv6ad= drs': >=3B ['fe80::a6ba:dbff:feec:eacd/64'=2C '2001:4830:1692:1:a6ba:dbff= :feec:eacd/64']=2C 'speed': 1000=2C 'mtu': '1500'}}<br>>=3B >=3B >=3B= >=3B <br>>=3B >=3B >=3B >=3B Based on that input=2C the engine s= ends setupNetwork command to configure the management network on top of 'em= 1' nic.<br>>=3B >=3B >=3B >=3B However=2C since it has no bootproto=
col or gateway=2C it is identified as bootproto=3DNONE=2C which result in e= ngine not to pass ip >=3B address/subnet/gateway to vdsm=2C therefore the= command fails.<br>>=3B >=3B >=3B <br>>=3B >=3B >=3B This seems= very similar to what triggered<br>>=3B >=3B >=3B <br>>=3B >=3B &= gt=3B Bug 987813 - [RFE] report BOOTPROTO and BONDING_OPTS independent of<= br>>=3B >=3B >=3B netdevice.cfg<br>>=3B >=3B >=3B <br>>=3B &= gt=3B >=3B Vdsm does not really cope with network definitions that are no= t<br>>=3B >=3B >=3B ifcfg-based. I do not know what makes Fedora 19 s= ometimes use ifcfg<br>>=3B >=3B >=3B out-of-the-box and sometimes not= =2C but in order to play well with the<br>>=3B >=3B >=3B current impl= ementation of Vdsm=2C we should make it use legacy initscript<br>>=3B >= =3B >=3B for network definition.<br>>=3B >=3B >=3B <br>>=3B >= =3B See the above. I wonder if netinstall.iso messes with this.<br>>=3B &= gt=3B >=3B >=3B <br>>=3B >=3B >=3B >=3B networks=3D[ovirtmgmt {= id=3Db05f272d-473f-41cb-b5a0-25cf06951f6b=2C description=3DManagement Netwo= rk=2C comment=3Dnull=2C subnet=3Dnull=2C gateway=3Dnull=2C >=3B type=3Dnu= ll=2C vlanId=3Dnull=2C stp=3Dfalse=2C dataCenterId=3D7a7e4a7a-ecc8-4b3e-b20= 0-2b54d0809a52=2C mtu=3D0=2C vmNetwork=3Dtrue=2C cluster=3DNetworkCluster {= id=3D>=3B {clusterId=3Dnull=2C networkId=3Dnull}=2C status=3DOPERATIONAL= =2C display=3Dfalse=2C required=3Dtrue=2C migration=3Dfalse}=2C providedBy= =3Dnull=2C label=3Dnull}]=2C<br>>=3B >=3B >=3B >=3B bonds=3D[]=2C<b= r>>=3B >=3B >=3B >=3B interfaces=3D[em1 {id=3D80a1e7c8-05bd-4209-a6= fe-a23c38538330=2C vdsId=3D5e3da5d9-7133-4119-86c0-f655d3e37b38=2C name=3De= m1=2C >=3B macAddress=3Da4:ba:db:ec:ea:cd=2C networkName=3Dovirtmgmt=2C b= ondName=3Dnull=2C bootProtocol=3DNONE=2C address=3D192.168.2.9=2C subnet=3D= 255.255.255.0=2C >=3B gateway=3Dnull=2C mtu=3D0=2C bridged=3Dtrue=2C spee= d=3D1000=2C type=3D2=2C networkImplementationDetails=3Dnull}]=2C<br>>=3B = >=3B >=3B >=3B removedNetworks=3D[]=2C<br>>=3B >=3B >=3B >=3B= <br>>=3B >=3B >=3B >=3B As received by VDSM=2C where we can see th= e ip address and the subnet are omitted:<br>>=3B >=3B >=3B >=3B Thr= ead-17::DEBUG::2013-09-23 07:07:21=2C314::BindingXMLRPC::979::vds::(wrapper= ) client [192.168.2.9]::call setupNetworks with >=3B ({'ovirtmgmt': {'nic= ': 'em1'=2C 'STP': 'no'=2C 'bridged': 'true'}}=2C {}=2C {'connectivityCheck= ': 'true'=2C 'connectivityTimeout': 120}) {}<br>>=3B >=3B >=3B >=3B= <br>>=3B >=3B >=3B >=3B Looking at the traceback on vdsm.log revea= ls less information:<br>>=3B >=3B >=3B >=3B storageRefresh::DEBUG::= 2013-09-23 07:07:22=2C597::misc::817::SamplingMethod::(__call__) Returning = last result<br>>=3B >=3B >=3B >=3B Thread-17::ERROR::2013-09-23 07:= 09:24=2C154::API::1261::vds::(setupNetworks)<br>>=3B >=3B >=3B >=3B= Traceback (most recent call last):<br>>=3B >=3B >=3B >=3B File "/u= sr/share/vdsm/API.py"=2C line 1259=2C in setupNetworks<br>>=3B >=3B >= =3B >=3B supervdsm.getProxy().setupNetworks(networks=2C bondings=2C optio= ns)<br>>=3B >=3B >=3B >=3B File "/usr/share/vdsm/supervdsm.py"=2C l= ine 50=2C in __call__<br>>=3B >=3B >=3B >=3B return callMethod()<br= >>=3B >=3B >=3B >=3B File "/usr/share/vdsm/supervdsm.py"=2C line 48= =2C in <=3Blambda>=3B<br>>=3B >=3B >=3B >=3B **kwargs)<br>>= =3B >=3B >=3B >=3B File "<=3Bstring>=3B"=2C line 2=2C in setupNet= works<br>>=3B >=3B >=3B >=3B File "/usr/lib64/python2.7/multiproces= sing/managers.py"=2C line 773=2C in _callmethod<br>>=3B >=3B >=3B >= =3B raise convert_to_error(kind=2C result)<br>>=3B >=3B >=3B >=3B C= onfigNetworkError: (29=2C '')<br>>=3B >=3B >=3B >=3B <br>>=3B >= =3B >=3B >=3B <br>>=3B >=3B >=3B >=3B Nicholas=2C could you add= also /var/log/vdsm/supervdsm.log and /var/log/messages so we can get more = input about<br>>=3B >=3B >=3B >=3B the failure to bring up em1 ?<br= >>=3B >=3B Yep=2C attached.<br>>=3B >=3B <br>>=3B <br= ><=3Bsnipped>=3B<br>>=3B <br>>=3B Here=2C Vdsm is trying to configu= re em1 with no ip address (because it<br>>=3B found no ifcfg-em1 to begin= with). But then=2C it fails to do so since<br>>=3B NetworkManager is sti= ll running.<br>>=3B <br>>=3B So if possible=2C make sure ifcfg-em1 exis= ts (and has the correct<br>>=3B BOOTPROT=3Ddhcp in it) and the NetworkMan= ager is off before initiating<br>>=3B installation. That's annoying=2C I = know. It should be fix=2C for sure. But<br>>=3B currently it is a must.<b= r>>=3B <br>>=3B Regards=2C<br>>=3B Dan.<br>Hopefully I didn't miss an= y other comments in that log snippet of log file ^^=3B=3B It's good to know= why it keeps failing. I'm just trying to figure out how to move forward fr= om here=2C and I'll take a crack at it this evening.<br>I thought that Netw= orkManager only needs to be disabled if you are using a static IP? I did tr= y disabling NM before I realized it said only for static and had a failure= =2C but probably because of the interface/ifcfg issue. I will try again thi= s evening.<br><br>I'll try to jump into IRC by 5pm EDT if you happen to be = around.<br><br></div></div>=0A= </div></body> </html>=
--_9b96a2b2-e7ae-4c34-91de-2f1f6da28755_--