----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Maurice James" <mjames@media-node.com>, fromani@redhat.com Cc: "Moti Asayag" <masayag@redhat.com>, "users" <users@ovirt.org>, "Antoni Segura Puimedon" <asegurap@redhat.com> Sent: Monday, July 21, 2014 6:49:56 PM Subject: Re: [ovirt-users] Logical network error
On Mon, Jul 21, 2014 at 07:35:00AM -0400, Maurice James wrote:
Here are the supervdsm logs
Hmm, it's a trove of errors and tracebacks; there's a lot to debug, for example, here Vdsm was asked to create a 'migration' network on top of bond0 that was already used by ovirtmgmt. Engine should have blocked that. Moti?
Obviously should have been blocked from engine side, however there are couple of issues with that migration network which might contempt the engine caps: MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,677::configNetwork::250::root::(addNetwork) validating network... MainProcess|Thread-43::INFO::2014-04-11 08:56:24,677::configNetwork::272::root::(addNetwork) Adding network migration with vlan=5, bonding=bond0, nics=['em1', 'em2'], bondingOptions=mode=4 miimon=100, mtu=None, bridged=False, defaultRoute=False,options={'implicitBonding': True} MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,679::ifcfg::447::root::(_persistentBackup) backing up ifcfg-bond0.5: # original file did not exist MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,681::ifcfg::313::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/ifcfg-bond0.5 (until next 'set safe config') MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,681::ifcfg::537::root::(writeConfFile) Writing to file /etc/sysconfig/network-scripts/ifcfg-bond0.5 configuration: # Generated by VDSM version 4.14.6-7.git41501b3.el6 DEVICE=bond0.5 ONBOOT=yes VLAN=yes DEFROUTE=no NM_CONTROLLED=no MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,684::utils::642::root::(execCmd) '/sbin/ip route show to 0.0.0.0/0 table all' (cwd None) MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,690::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,728::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.2' (cwd None) MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,735::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,741::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.4' (cwd None) MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,747::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,753::utils::642::root::(execCmd) '/sbin/ip -d link show dev bond0.3' (cwd None) MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,759::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,764::__init__::122::root::(_addSourceRoute) Adding source route bond0.5, None, None, None MainProcess|Thread-43::ERROR::2014-04-11 08:56:24,764::sourceRoute::68::root::(configure) ipaddr, mask or gateway not received MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,764::utils::642::root::(execCmd) '/sbin/ifup bond0.5' (cwd None) MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,914::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,917::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 43 edom: 19 level: 2 message: Network not found: no network with matching name 'vdsm-migration' MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,918::ifcfg::324::root::(_atomicNetworkBackup) Backed up migration MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,919::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 43 edom: 19 level: 2 message: Network not found: no network with matching name 'vdsm-migration' MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,919::ifcfg::333::root::(_persistentNetworkBackup) backing up network migration: # original file did not exist MainProcess|Thread-43::DEBUG::2014-04-11 08:56:24,921::ifcfg::313::root::(writeBackupFile) Persistently backed up /var/lib/vdsm/netconfback/logicalnetworks/migration (until next 'set safe config') MainProcess|Thread-43::DEBUG::2014-04-11 08:56:25,031::configNetwork::651::setupNetworks::(setupNetworks) Checking connectivity... MainProcess|Thread-43::DEBUG::2014-04-11 08:56:26,035::supervdsmServer::103::SuperVdsm.ServerCallback::(wrapper) return setupNetworks with None MainProcess|Thread-43::DEBUG::201 I wasn't able to find the output of getCapabilities for the host in the logs. Without 'getCaps' it is hard to know which network configuration was reported to the engine by vdsm. But if 'vdsm-migration' is not a know network (based on above message), the engine might allow to use the underlying interface. In addition, there are a lot of setup-networks call which most of them failed due to 'resource unavailable'.
MainProcess|Thread-47826::DEBUG::2014-04-11 10:00:26,335::configNetwork::589::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'migration': {'bonding': 'bond0', 'bridged': 'false'}}, bondings:{}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 120} Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 98, in wrapper res = func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 202, in setupNetworks return configNetwork.setupNetworks(networks, bondings, **options) File "/usr/share/vdsm/configNetwork.py", line 648, in setupNetworks implicitBonding=True, **d) File "/usr/share/vdsm/configNetwork.py", line 186, in wrapped return func(*args, **kwargs) File "/usr/share/vdsm/configNetwork.py", line 256, in addNetwork bridged) File "/usr/share/vdsm/configNetwork.py", line 170, in _validateInterNetworkCompatibility _validateNoDirectNet(ifaces_bridgeless) File "/usr/share/vdsm/configNetwork.py", line 154, in _validateNoDirectNet (iface, iface_net)) ConfigNetworkError: (21, "interface 'bond0' already member of network 'ovirtmgmt'")
There are also repeated failed attempts to create a payload disk: did you notice when they happen? Could it be related to insufficient disk space?
MainProcess|clientIFinit::ERROR::2014-03-25 22:13:02,529::supervdsmServer::100::SuperVdsm.ServerCallback::(wrapper) Error in mkFloppyFs Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 98, in wrapper res = func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 325, in mkFloppyFs return mkimage.mkFloppyFs(vmId, files, volId) File "/usr/share/vdsm/mkimage.py", line 104, in mkFloppyFs "code %s, out %s\nerr %s" % (rc, out, err)) OSError: [Errno 5] could not create floppy file: code 1, out mkfs.msdos 3.0.9 (31 Jan 2010)
err mkfs.msdos: unable to create /var/run/vdsm/payload/94632c2e-28a0-4304-9261-c302e0027604.ecac527e731a2a0057dc6a3ae3df9ba3.img
In any case, if you manage to reproduce these issues, please open a detailed bug entry.
Regards, Dan.