Re: [Users] Unable to finish AIO 3.3.0 - VDSM

Monday, 23 September 2013

I have looked at the getVdsCapabilities reported by VDSM for the first time, on which the
engine based its
setupNetwork command for configuring the management network:

'lastClientIface': 'em1',
'nics': {'em1': {'netmask': '255.255.255.0',
'addr': '192.168.2.9', 'hwaddr': 'a4:ba:db:ec:ea:cd',
'cfg': {}, 'ipv6addrs': ['fe80::a6ba:dbff:feec:eacd/64',
'2001:4830:1692:1:a6ba:dbff:feec:eacd/64'], 'speed': 1000, 'mtu':
'1500'}}

Based on that input, the engine sends setupNetwork command to configure the management
network on top of 'em1' nic.
However, since it has no bootprotocol or gateway, it is identified as bootproto=NONE,
which result in engine not to pass ip address/subnet/gateway to vdsm, therefore the
command fails.

networks=[ovirtmgmt {id=b05f272d-473f-41cb-b5a0-25cf06951f6b, description=Management
Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false,
dataCenterId=7a7e4a7a-ecc8-4b3e-b200-2b54d0809a52, mtu=0, vmNetwork=true,
cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL,
display=false, required=true, migration=false}, providedBy=null, label=null}],
	bonds=[],
	interfaces=[em1 {id=80a1e7c8-05bd-4209-a6fe-a23c38538330,
vdsId=5e3da5d9-7133-4119-86c0-f655d3e37b38, name=em1, macAddress=a4:ba:db:ec:ea:cd,
networkName=ovirtmgmt, bondName=null, bootProtocol=NONE, address=192.168.2.9,
subnet=255.255.255.0, gateway=null, mtu=0, bridged=true, speed=1000, type=2,
networkImplementationDetails=null}],
	removedNetworks=[],

As received by VDSM, where we can see the ip address and the subnet are omitted:
Thread-17::DEBUG::2013-09-23 07:07:21,314::BindingXMLRPC::979::vds::(wrapper) client
[192.168.2.9]::call setupNetworks with ({'ovirtmgmt': {'nic':
'em1', 'STP': 'no', 'bridged': 'true'}}, {},
{'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}

Looking at the traceback on vdsm.log reveals less information:
storageRefresh::DEBUG::2013-09-23 07:07:22,597::misc::817::SamplingMethod::(__call__)
Returning last result
Thread-17::ERROR::2013-09-23 07:09:24,154::API::1261::vds::(setupNetworks)
Traceback (most recent call last):
  File "/usr/share/vdsm/API.py", line 1259, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (29, '')

Nicholas, could you add also /var/log/vdsm/supervdsm.log and /var/log/messages so we can
get more input about
the failure to bring up em1 ?

----- Original Message -----
...
 From: "Alon Bar-Lev" <alonbl(a)redhat.com&gt;
 To: "Nicholas Kesick" <cybertimber2000(a)hotmail.com&gt;, "Moti
Asayag" <masayag(a)redhat.com&gt;
 Cc: "oVirt Mailing List" <users(a)ovirt.org&gt;
 Sent: Monday, September 23, 2013 5:36:57 PM
 Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM

 Moti,

 Any hint?

 Thanks,
 Alon

 ---
 2013-09-23 07:09:26,813 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-6-thread-3)
 Command PollVDS execution failed. Exception: RuntimeException:
 java.util.concurrent.ExecutionException:
 java.lang.reflect.InvocationTargetException
 2013-09-23 07:09:27,314 WARN
 [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
 (pool-6-thread-3) Unexpected return value: StatusForXmlRpc [mCode=29,
 mMessage=]
 2013-09-23 07:09:27,314 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
 (pool-6-thread-3) Failed in SetupNetworksVDS method
 2013-09-23 07:09:27,314 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
 (pool-6-thread-3) Error code unexpected and error message
 VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error =
 2013-09-23 07:09:27,315 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
 (pool-6-thread-3)
 org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
 VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error =
 2013-09-23 07:09:27,315 ERROR
 [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand]
 (pool-6-thread-3) Command SetupNetworksVDS execution failed. Exception:
 VDSErrorException: VDSGenericException: VDSErrorException: Failed to
 SetupNetworksVDS, error =
 2013-09-23 07:09:27,315 ERROR
 [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand]
 (pool-6-thread-3) Command
 org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw Vdc Bll
 exception. With error message VdcBLLException:
 org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
 VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error =
 (Failed with VDSM error unexpected and code 16)
 2013-09-23 07:09:27,340 INFO
 [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
 (pool-6-thread-3) Failed to configure management network: Failed to
 configure management network on host local_host due to setup networks
 failure.
 2013-09-23 07:09:27,340 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand]
 (pool-6-thread-3) Host installation failed for host
 5e3da5d9-7133-4119-86c0-f655d3e37b38, local_host.:
 org.ovirt.engine.core.bll.InstallVdsCommand$VdsInstallException: Failed to
 configure management network on the host
 	at

	org.ovirt.engine.core.bll.InstallVdsCommand.configureManagementNetwork(InstallVdsCommand.java:306)
 	[bll.jar:]
 	at
 	org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:205)
 	[bll.jar:]
 	at
 	org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:105)
 	[bll.jar:]
 	at
 	org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1128)
 	[bll.jar:]
 	at

	org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1213)
 	[bll.jar:]
 	at
 	org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1871)
 	[bll.jar:]
 	at

	org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174)
 	[utils.jar:]
 	at

	org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116)
 	[utils.jar:]
 	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1231)
 	[bll.jar:]
 	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:365)
 	[bll.jar:]
 	at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:408) [bll.jar:]
 	at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:387)
 	[bll.jar:]
 	at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:627)
 	[bll.jar:]
 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 	[rt.jar:1.7.0_40]
 	at
 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 	[rt.jar:1.7.0_40]
 	at
 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 	[rt.jar:1.7.0_40]
 	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_40]
 	at

	org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53)
 	[jboss-invocation.jar:1.1.1.Final]
 	at

	org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
 	[jboss-invocation.jar:1.1.1.Final]
 	at

	org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at

	org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53)
 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:211)
 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:363)
 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:194)
 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at

	org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at

	org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59)
 	[jboss-as-ejb3-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at

	org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
 	[jboss-invocation.jar:1.1.1.Final]
 	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:173)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
 	[jboss-invocation.jar:1.1.1.Final]
 	at
 	org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72)
 	[jboss-as-ee-7.1.1.Final.jar:7.1.1.Final]
 	at
 	org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view10.runInternalAction(Unknown
 	Source) [bll.jar:]
 	at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:173)
 	[bll.jar:]
 	at

	org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71)
 	[utils.jar:]
 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 	[rt.jar:1.7.0_40]
 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
 	[rt.jar:1.7.0_40]
 	at
 	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 	[rt.jar:1.7.0_40]
 	at
 	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 	[rt.jar:1.7.0_40]
 	at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]
 Caused by:
 org.ovirt.engine.core.bll.network.NetworkConfigurator$NetworkConfiguratorException:
 Failed to configure management network
 	at

	org.ovirt.engine.core.bll.network.NetworkConfigurator.configureManagementNetwork(NetworkConfigurator.java:241)
 	[bll.jar:]
 	at

	org.ovirt.engine.core.bll.network.NetworkConfigurator.createManagementNetworkIfRequired(NetworkConfigurator.java:69)
 	[bll.jar:]
 	at

	org.ovirt.engine.core.bll.InstallVdsCommand.configureManagementNetwork(InstallVdsCommand.java:298)
 	[bll.jar:]
 	... 54 more

 2013-09-23 07:09:27,351 INFO
 [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-6-thread-3)
 START, SetVdsStatusVDSCommand(HostName = local_host, HostId =
 5e3da5d9-7133-4119-86c0-f655d3e37b38, status=NonOperational,
 nonOperationalReason=NONE), log id: 478baa54
 2013-09-23 07:09:27,356 INFO
 [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-6-thread-3)
 FINISH, SetVdsStatusVDSCommand, log id: 478baa54
 2013-09-23 07:09:27,373 INFO
 [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
 (pool-6-thread-3) Correlation ID: 35dad8c0, Job ID:
 e8796c44-de74-4b42-925d-93397f54be92, Call Stack: null, Custom Event ID: -1,
 Message: Host local_host installation failed. Failed to configure management
 network on the host.
 2013-09-23 07:09:27,407 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand]
 (pool-6-thread-3) Lock freed to object EngineLock [exclusiveLocks= key:
 5e3da5d9-7133-4119-86c0-f655d3e37b38 value: VDS
 , sharedLocks= ]

 ----- Original Message -----
 > From: "Nicholas Kesick" <cybertimber2000(a)hotmail.com&gt;
 > To: "Alon Bar-Lev" <alonbl(a)redhat.com&gt;
 > Cc: "oVirt Mailing List" <users(a)ovirt.org&gt;
 > Sent: Monday, September 23, 2013 6:18:29 PM
 > Subject: RE: [Users] Unable to finish AIO 3.3.0 - VDSM
 > 
 > 
 > 
 > > Date: Mon, 23 Sep 2013 11:09:23 -0400
 > > From: alonbl(a)redhat.com
 > > To: cybertimber2000(a)hotmail.com
 > > CC: users(a)ovirt.org
 > > Subject: Re: [Users] Unable to finish AIO 3.3.0 - VDSM
 > > 
 > > 
 > > 
 > > ----- Original Message -----
 > > > From: "Nicholas Kesick" <cybertimber2000(a)hotmail.com&gt;
 > > > To: "oVirt Mailing List" <users(a)ovirt.org&gt;
 > > > Sent: Monday, September 23, 2013 5:52:17 PM
 > > > Subject: [Users] Unable to finish AIO 3.3.0 - VDSM
 > > > 
 > > > I'm having trouble with the AIO 3.3.0 install on Fedora 19 using
 > > > jbrooks
 > > > directions - I have DHCP, so I omitted the NM portion.
 > > > The good news is that the engine-setup completes just fine, but the
 > > > VDSM
 > > > portion fails. I thought it had to do with a missing dependency but so
 > > > far
 > > > that hasn't helped either. Does anyone have any suggestions?
 > > > 
 > > > The install is a Fedora 19 "minimal" install with the
"standard"
 > > > package
 > > > of
 > > > add-ons. I've tried with and without SELINUX in permissive. The only
 > > > thing I
 > > > have noticed is that the ovirtmgmt network is missing. When I try to
 > > > activate the host using the web interface ovirtmgmt appears, but
 > > > without
 > > > an
 > > > IP address, so the entire system loses connectivity.
 > > > 
 > > > ovirt-engine-setup and vdsm log are attached and compressed.
 > > 
 > > Please attach any log you may have at /var/log/ovirt-engine/engine.log
 > > /var/log/ovirt-engine/host-deploy/*.log
 > > 
 > Attached. Looking through them myself now as well.
 > > > 
 > > > 
 > > > 
 > > > 
 > > > 
 > > > _______________________________________________
 > > > Users mailing list
 > > > Users(a)ovirt.org
 > > > http://lists.ovirt.org/mailman/listinfo/users
 > > > 
 > 

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [Users] Unable to finish AIO 3.3.0 - VDSM