[Users] Unable to finish AIO 3.3.0 - VDSM

Moti Asayag masayag at redhat.com
Mon Sep 23 18:25:49 UTC 2013 

I have looked at the getVdsCapabilities reported by VDSM for the first time, on which the engine based its
setupNetwork command for configuring the management network:

'lastClientIface': 'em1',
'nics': {'em1': {'netmask': '255.255.255.0', 'addr': '192.168.2.9', 'hwaddr': 'a4:ba:db:ec:ea:cd', 'cfg': {}, 'ipv6addrs': ['fe80::a6ba:dbff:feec:eacd/64', '2001:4830:1692:1:a6ba:dbff:feec:eacd/64'], 'speed': 1000, 'mtu': '1500'}}

Based on that input, the engine sends setupNetwork command to configure the management network on top of 'em1' nic.
However, since it has no bootprotocol or gateway, it is identified as bootproto=NONE, which result in engine not to pass ip address/subnet/gateway to vdsm, therefore the command fails.

networks=[ovirtmgmt {id=b05f272d-473f-41cb-b5a0-25cf06951f6b, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=7a7e4a7a-ecc8-4b3e-b200-2b54d0809a52, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true, migration=false}, providedBy=null, label=null}],
	bonds=[],
	interfaces=[em1 {id=80a1e7c8-05bd-4209-a6fe-a23c38538330, vdsId=5e3da5d9-7133-4119-86c0-f655d3e37b38, name=em1, macAddress=a4:ba:db:ec:ea:cd, networkName=ovirtmgmt, bondName=null, bootProtocol=NONE, address=192.168.2.9, subnet=255.255.255.0, gateway=null, mtu=0, bridged=true, speed=1000, type=2, networkImplementationDetails=null}],
	removedNetworks=[],

As received by VDSM, where we can see the ip address and the subnet are omitted:
Thread-17::DEBUG::2013-09-23 07:07:21,314::BindingXMLRPC::979::vds::(wrapper) client [192.168.2.9]::call setupNetworks with ({'ovirtmgmt': {'nic': 'em1', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}

Looking at the traceback on vdsm.log reveals less information:
storageRefresh::DEBUG::2013-09-23 07:07:22,597::misc::817::SamplingMethod::(__call__) Returning last result
Thread-17::ERROR::2013-09-23 07:09:24,154::API::1261::vds::(setupNetworks)
Traceback (most recent call last):
  File "/usr/share/vdsm/API.py", line 1259, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (29, '')


Nicholas, could you add also /var/log/vdsm/supervdsm.log and /var/log/messages so we can get more input about
the failure to bring up em1 ?

----- Original Message -----
> From: "Alon Bar-Lev" <alonbl at redhat.com>
> To: "Nicholas Kesick" <cybertimber2000 at hotmail.com>, "Moti Asayag" <masayag at redhat.com>
> Cc: "oVirt Mailing List" <users at ovirt.org>
> Sent: Monday, September 23, 2013 5:36:57 PM
> Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM
> 
> 
> Moti,
> 
> Any hint?
> 
> Thanks,
> Alon
> 
> ---
> 2013-09-23 07:09:26,813 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-6-thread-3)
> Command PollVDS execution failed. Exception: RuntimeException:
> java.util.concurrent.ExecutionException:
> java.lang.reflect.InvocationTargetException
> 2013-09-23 07:09:27,314 WARN
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
> (pool-6-thread-3) Unexpected return value: StatusForXmlRpc [mCode=29,
> mMessage=]
> 2013-09-23 07:09:27,314 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
> (pool-6-thread-3) Failed in SetupNetworksVDS method
> 2013-09-23 07:09:27,314 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
> (pool-6-thread-3) Error code unexpected and error message
> VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error =
> 2013-09-23 07:09:27,315 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
> (pool-6-thread-3)
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error =
> 2013-09-23 07:09:27,315 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
> (pool-6-thread-3) Command SetupNetworksVDS execution failed. Exception:
> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
> SetupNetworksVDS, error =
> 2013-09-23 07:09:27,315 ERROR
> [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand]
> (pool-6-thread-3) Command
> org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw Vdc Bll
> exception. With error message VdcBLLException:
> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
> VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error =
> (Failed with VDSM error unexpected and code 16)
> 2013-09-23 07:09:27,340 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (pool-6-thread-3) Failed to configure management network: Failed to
> configure management network on host local_host due to setup networks
> failure.
> 2013-09-23 07:09:27,340 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand]
> (pool-6-thread-3) Host installation failed for host
> 5e3da5d9-7133-4119-86c0-f655d3e37b38, local_host.:
> org.ovirt.engine.core.bll.InstallVdsCommand$VdsInstallException: Failed to
> configure management network on the host
> 	at
> 	org.ovirt.engine.core.bll.InstallVdsCommand.configureManagementNetwork(InstallVdsCommand.java:306)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:205)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:105)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1128)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1213)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1871)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
> 	[utils.jar:]
> 	at
> 	org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
> 	[utils.jar:]
> 	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1231)
> 	[bll.jar:]
> 	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:365)
> 	[bll.jar:]
> 	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:408) [bll.jar:]
> 	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:387)
> 	[bll.jar:]
> 	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:627)
> 	[bll.jar:]
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	[rt.jar:1.7.0_40]
> 	at
> 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	[rt.jar:1.7.0_40]
> 	at
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	[rt.jar:1.7.0_40]
> 	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_40]
> 	at
> 	org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
> 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211)
> 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
> 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363)
> 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194)
> 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
> 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> 	[jboss-invocation.jar:1.1.1.Final]
> 	at
> 	org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
> 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
> 	at
> 	org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view10.runInternalAction(Unknown
> 	Source) [bll.jar:]
> 	at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:173)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71)
> 	[utils.jar:]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	[rt.jar:1.7.0_40]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	[rt.jar:1.7.0_40]
> 	at
> 	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	[rt.jar:1.7.0_40]
> 	at
> 	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	[rt.jar:1.7.0_40]
> 	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]
> Caused by:
> org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException:
> Failed to configure management network
> 	at
> 	org.ovirt.engine.core.bll.network.NetworkConfigurator.configureManagementNetwork(NetworkConfigurator.java:241)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:69)
> 	[bll.jar:]
> 	at
> 	org.ovirt.engine.core.bll.InstallVdsCommand.configureManagementNetwork(InstallVdsCommand.java:298)
> 	[bll.jar:]
> 	... 54 more
> 
> 2013-09-23 07:09:27,351 INFO
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-6-thread-3)
> START, SetVdsStatusVDSCommand(HostName = local_host, HostId =
> 5e3da5d9-7133-4119-86c0-f655d3e37b38, status=NonOperational,
> nonOperationalReason=NONE), log id: 478baa54
> 2013-09-23 07:09:27,356 INFO
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-6-thread-3)
> FINISH, SetVdsStatusVDSCommand, log id: 478baa54
> 2013-09-23 07:09:27,373 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (pool-6-thread-3) Correlation ID: 35dad8c0, Job ID:
> e8796c44-de74-4b42-925d-93397f54be92, Call Stack: null, Custom Event ID: -1,
> Message: Host local_host installation failed. Failed to configure management
> network on the host.
> 2013-09-23 07:09:27,407 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand]
> (pool-6-thread-3) Lock freed to object EngineLock [exclusiveLocks= key:
> 5e3da5d9-7133-4119-86c0-f655d3e37b38 value: VDS
> , sharedLocks= ]
> 
> 
> ----- Original Message -----
> > From: "Nicholas Kesick" <cybertimber2000 at hotmail.com>
> > To: "Alon Bar-Lev" <alonbl at redhat.com>
> > Cc: "oVirt Mailing List" <users at ovirt.org>
> > Sent: Monday, September 23, 2013 6:18:29 PM
> > Subject: RE: [Users] Unable to finish AIO 3.3.0 - VDSM
> > 
> > 
> > 
> > > Date: Mon, 23 Sep 2013 11:09:23 -0400
> > > From: alonbl at redhat.com
> > > To: cybertimber2000 at hotmail.com
> > > CC: users at ovirt.org
> > > Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Nicholas Kesick" <cybertimber2000 at hotmail.com>
> > > > To: "oVirt Mailing List" <users at ovirt.org>
> > > > Sent: Monday, September 23, 2013 5:52:17 PM
> > > > Subject: [Users] Unable to finish AIO 3.3.0 - VDSM
> > > > 
> > > > I'm having trouble with the AIO 3.3.0 install on Fedora 19 using
> > > > jbrooks
> > > > directions - I have DHCP, so I omitted the NM portion.
> > > > The good news is that the engine-setup completes just fine, but the
> > > > VDSM
> > > > portion fails. I thought it had to do with a missing dependency but so
> > > > far
> > > > that hasn't helped either. Does anyone have any suggestions?
> > > > 
> > > > The install is a Fedora 19 "minimal" install with the "standard"
> > > > package
> > > > of
> > > > add-ons. I've tried with and without SELINUX in permissive. The only
> > > > thing I
> > > > have noticed is that the ovirtmgmt network is missing. When I try to
> > > > activate the host using the web interface ovirtmgmt appears, but
> > > > without
> > > > an
> > > > IP address, so the entire system loses connectivity.
> > > > 
> > > > ovirt-engine-setup and vdsm log are attached and compressed.
> > > 
> > > Please attach any log you may have at /var/log/ovirt-engine/engine.log
> > > /var/log/ovirt-engine/host-deploy/*.log
> > > 
> > Attached. Looking through them myself now as well.
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > _______________________________________________
> > > > Users mailing list
> > > > Users at ovirt.org
> > > > http://lists.ovirt.org/mailman/listinfo/users
> > > > 
> > 
>

More information about the Users mailing list