On Thu, Nov 15, 2012 at 5:34 AM, Roy Golan <rgolan@redhat.com> wrote:

logic/flow aside, still the error is on engine side - validating the VdsNetworkInterface entity which
seems to be missing an "interfaces" field or method thus throwing this runtime error.
I wonder if GWT serialized something unexpected here.
Alan - can you try invoking setup networks and see if any other action is terminated the same way?

Yes, no matter what I do on Setup Host Networks, I get the same error when I click OK. Even if I make not changes at all, I get the same when I click OK. I have to Cancel to get out of that screen.

I have just run though the same process on a different engine controlling a single node. Not changing anything on the Setup Host Networks screen, it closes successfully when I click OK. Trying to add the sandbox network to em1 gives the error "Error while executing action Setup Networks: Internal oVirt Engine Error". This shows up in engine.log:

2012-11-15 12:04:41,497 INFO [org.ovirt.engine.core.bll.SetupNetworksCommand] (ajp--0.0.0.0-8009-2) [51a6bccb] Running command: SetupNetworksCommand internal: false. Entities affected : ID: 65f7ba84-2aa5-11e2-a193-001a4aa8770c Type: VDS

2012-11-15 12:04:41,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--0.0.0.0-8009-2) [51a6bccb] START, SetupNetworksVDSCommand(vdsId = 65f7ba84-2aa5-11e2-a193-001a4aa8770c), log id: 1e23f24a

2012-11-15 12:04:41,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--0.0.0.0-8009-2) [51a6bccb] FINISH, SetupNetworksVDSCommand, log id: 1e23f24a

2012-11-15 12:04:42,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] Failed in SetupNetworksVDS method

2012-11-15 12:04:42,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] Error code ERR_USED_NIC and error message VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Setup attached more than one network to nic em1, some of which aren't vlans

2012-11-15 12:04:42,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Setup attached more than one network to nic em1, some of which aren't vlans

2012-11-15 12:04:42,018 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] Command SetupNetworksVDS execution failed. Exception: RuntimeException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Setup attached more than one network to nic em1, some of which aren't vlans

2012-11-15 12:04:42,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--0.0.0.0-8009-2) [51a6bccb] No string for UNASSIGNED type. Use default Log

And vdsm.log show this:

Dummy-3930::DEBUG::2012-11-15 12:04:41,618::__init__::1164::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/88c14568-2860-11e2-8ecf-001a4aa8770c/mastersd/dom_md/inbox iflag=direct,

fullblock count=1 bs=1024000' (cwd None)

Dummy-3930::DEBUG::2012-11-15 12:04:41,721::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0531377 s, 19

.3 MB/s\n'; <rc> = 0

Thread-397033::DEBUG::2012-11-15 12:04:43,084::BindingXMLRPC::859::vds::(wrapper) client [192.168.119.232]::call setupNetworks with ({'ovirtmgmt': {'nic': 'em1', 'bridged': 'true'}, 'sa

ndbox': {'nic': 'em1', 'vlan': '2', 'bridged': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 60000}) {} flowID [51a6bccb]

Thread-397034::DEBUG::2012-11-15 12:04:43,085::BindingXMLRPC::859::vds::(wrapper) client [192.168.119.232]::call ping with () {} flowID [51a6bccb]

Thread-397034::DEBUG::2012-11-15 12:04:43,086::BindingXMLRPC::865::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}

MainProcess|Thread-397033::DEBUG::2012-11-15 12:04:43,099::configNetwork::1083::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'ni

c': 'em1', 'bridged': 'true'}, 'sandbox': {'nic': 'em1', 'vlan': '2', 'bridged': 'true'}}, bondings:{}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 60000}

MainProcess|Thread-397033::DEBUG::2012-11-15 12:04:43,099::configNetwork::1087::root::(setupNetworks) Validating configuration

MainProcess|Thread-397033::ERROR::2012-11-15 12:04:43,110::configNetwork::1155::setupNetworks::(setupNetworks) (24, "Setup attached more than one network to nic em1, some of which aren'

t vlans")

Traceback (most recent call last):

File "/usr/share/vdsm/configNetwork.py", line 1088, in setupNetworks

_validateNetworkSetup(dict(networks), dict(bondings))

File "/usr/share/vdsm/configNetwork.py", line 969, in _validateNetworkSetup

"Setup attached more than one network to nic %s, some of which aren't vlans"%(nic))

ConfigNetworkError: (24, "Setup attached more than one network to nic em1, some of which aren't vlans")

MainProcess|Thread-397033::ERROR::2012-11-15 12:04:43,112::supervdsmServer::76::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks

Traceback (most recent call last):

File "/usr/share/vdsm/supervdsmServer.py", line 74, in wrapper

return func(*args, **kwargs)

File "/usr/share/vdsm/supervdsmServer.py", line 150, in setupNetworks

return configNetwork.setupNetworks(networks, bondings, **options)

File "/usr/share/vdsm/configNetwork.py", line 1088, in setupNetworks

_validateNetworkSetup(dict(networks), dict(bondings))

File "/usr/share/vdsm/configNetwork.py", line 969, in _validateNetworkSetup

"Setup attached more than one network to nic %s, some of which aren't vlans"%(nic))

ConfigNetworkError: (24, "Setup attached more than one network to nic em1, some of which aren't vlans")

Thread-397033::ERROR::2012-11-15 12:04:43,112::API::1158::vds::(setupNetworks) Setup attached more than one network to nic em1, some of which aren't vlans

Traceback (most recent call last):

File "/usr/share/vdsm/API.py", line 1156, in setupNetworks

supervdsm.getProxy().setupNetworks(networks, bondings, options)

File "/usr/share/vdsm/supervdsm.py", line 67, in __call__

return callMethod()

File "/usr/share/vdsm/supervdsm.py", line 65, in <lambda>

**kwargs)

File "<string>", line 2, in setupNetworks

File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod

raise convert_to_error(kind, result)

ConfigNetworkError: (24, "Setup attached more than one network to nic em1, some of which aren't vlans")

Thread-397033::DEBUG::2012-11-15 12:04:43,113::BindingXMLRPC::865::vds::(wrapper) return setupNetworks with {'status': {'message': "Setup attached more than one network to nic em1, some of which aren't vlans", 'code': 24}}

Dummy-3930::DEBUG::2012-11-15 12:04:43,732::__init__::1164::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/88c14568-2860-11e2-8ecf-001a4aa8770c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)

Dummy-3930::DEBUG::2012-11-15 12:04:43,841::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0565243 s, 18.1 MB/s\n'; <rc> = 0

VM Channels Listener::DEBUG::2012-11-15 12:04:45,065::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 317.

Dummy-3930::DEBUG::2012-11-15 12:04:45,852::__init__::1164::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/88c14568-2860-11e2-8ecf-001a4aa8770c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)

Dummy-3930::DEBUG::2012-11-15 12:04:45,960::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0565389 s, 18.1 MB/s\n'; <rc> = 0

[A side note: is it wise to keep DEBUG on VDSM logs? If not, is there an easy way to shut it off? That is an awful lot of traffic and might explain why my nodes ran so slow when I was booting off of USB sticks.]

So, all that seems to be consistent with Igor's comments and I'll have to do some fancy dancing to get this setup once I get past the other bug. It would be nice if the GUI relayed the error "Setup attached more than one network to nic em1, some of which aren't vlans" from VDSM to the user, or some clearer translation, but it sounds like that is on the way based on Moti's comments about the nightly's, so I'll let it drop.

This engine and node were built last week and the process went much more smoothly than this first engine having problems. For the problematic engine, which I setup a couple of months ago, the CentOS6 packages had bunch of broken symlinks to jars that were on the system, but not where the symlinks expected them to be. That appears to have been fixed as of least week when I setup this second engine. To fix it on the first, I manually created symlinks where the broken ones were pointed rather than messing with any symlinks created by the packages.

I wonder now if the packages on the bad engine (which are up to date) would have created different symlinks where I did and mine are blocking them from being created. I can rebuild the engine server if needed, but I would appreciate if some one can confirm that this error is consistent with a wrong version of a jar or some similar potential issue. Of course, I would also appreciate if there is a simpler solution. =)