On Thu, Nov 15, 2012 at 9:46 AM, Alan Johnson <span dir="ltr"><<a href="mailto:alan@datdec.com" target="_blank">alan@datdec.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Thu, Nov 15, 2012 at 5:34 AM, Roy Golan <span dir="ltr"><<a href="mailto:rgolan@redhat.com" target="_blank">rgolan@redhat.com</a>></span> wrote:<br></div><div class="gmail_quote"><div class="im">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>logic/flow aside, still the error is on engine side - validating the VdsNetworkInterface entity which</div></div>
seems to be missing an "interfaces" field or method thus throwing this runtime error.<br>
I wonder if GWT serialized something unexpected here.<br>
Alan - can you try invoking setup networks and see if any other action is terminated the same way?</blockquote><div><br></div></div><div>Yes, no matter what I do on Setup Host Networks, I get the same error when I click OK. Even if I make not changes at all, I get the same when I click OK. I have to Cancel to get out of that screen.</div>
</div>
</blockquote></div><br><div>I have just run though the same process on a different engine controlling a single node. Not changing anything on the Setup Host Networks screen, it closes successfully when I click OK. Trying to add the sandbox network to em1 gives the error "Error while executing action Setup Networks: Internal oVirt Engine Error". This shows up in engine.log:</div>
<div><div><font face="courier new, monospace">2012-11-15 12:04:41,497 INFO [org.ovirt.engine.core.bll.SetupNetworksCommand] (ajp--0.0.0.0-8009-2) [51a6bccb] Running command: SetupNetworksCommand internal: false. Entities affected : ID: 65f7ba84-2aa5-11e2-a193-001a4aa8770c Type: VDS</font></div>
<div><font face="courier new, monospace">2012-11-15 12:04:41,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--0.0.0.0-8009-2) [51a6bccb] START, SetupNetworksVDSCommand(vdsId = 65f7ba84-2aa5-11e2-a193-001a4aa8770c), log id: 1e23f24a</font></div>
<div><font face="courier new, monospace">2012-11-15 12:04:41,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--0.0.0.0-8009-2) [51a6bccb] FINISH, SetupNetworksVDSCommand, log id: 1e23f24a</font></div>
<div><font face="courier new, monospace">2012-11-15 12:04:42,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] Failed in SetupNetworksVDS method</font></div><div><font face="courier new, monospace">2012-11-15 12:04:42,017 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] Error code ERR_USED_NIC and error message VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Setup attached more than one network to nic em1, some of which aren't vlans</font></div>
<div><font face="courier new, monospace">2012-11-15 12:04:42,018 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Setup attached more than one network to nic em1, some of which aren't vlans</font></div>
<div><font face="courier new, monospace">2012-11-15 12:04:42,018 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--0.0.0.0-8009-2) [51a6bccb] Command SetupNetworksVDS execution failed. Exception: RuntimeException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Setup attached more than one network to nic em1, some of which aren't vlans</font></div>
<div><font face="courier new, monospace">2012-11-15 12:04:42,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--0.0.0.0-8009-2) [51a6bccb] No string for UNASSIGNED type. Use default Log</font></div>
</div><div><br></div><div>And vdsm.log show this:</div><div><div><font face="courier new, monospace">Dummy-3930::DEBUG::2012-11-15 12:04:41,618::__init__::1164::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/88c14568-2860-11e2-8ecf-001a4aa8770c/mastersd/dom_md/inbox iflag=direct,</font></div>
<div><font face="courier new, monospace">fullblock count=1 bs=1024000' (cwd None)</font></div><div><font face="courier new, monospace">Dummy-3930::DEBUG::2012-11-15 12:04:41,721::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0531377 s, 19</font></div>
<div><font face="courier new, monospace">.3 MB/s\n'; <rc> = 0</font></div><div><font face="courier new, monospace">Thread-397033::DEBUG::2012-11-15 12:04:43,084::BindingXMLRPC::859::vds::(wrapper) client [192.168.119.232]::call setupNetworks with ({'ovirtmgmt': {'nic': 'em1', 'bridged': 'true'}, 'sa</font></div>
<div><font face="courier new, monospace">ndbox': {'nic': 'em1', 'vlan': '2', 'bridged': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 60000}) {} flowID [51a6bccb]</font></div>
<div><font face="courier new, monospace">Thread-397034::DEBUG::2012-11-15 12:04:43,085::BindingXMLRPC::859::vds::(wrapper) client [192.168.119.232]::call ping with () {} flowID [51a6bccb]</font></div><div><font face="courier new, monospace">Thread-397034::DEBUG::2012-11-15 12:04:43,086::BindingXMLRPC::865::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}</font></div>
<div><font face="courier new, monospace">MainProcess|Thread-397033::DEBUG::2012-11-15 12:04:43,099::configNetwork::1083::setupNetworks::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'ni</font></div>
<div><font face="courier new, monospace">c': 'em1', 'bridged': 'true'}, 'sandbox': {'nic': 'em1', 'vlan': '2', 'bridged': 'true'}}, bondings:{}, options:{'connectivityCheck': 'true', 'connectivityTimeout': 60000}</font></div>
<div><font face="courier new, monospace">MainProcess|Thread-397033::DEBUG::2012-11-15 12:04:43,099::configNetwork::1087::root::(setupNetworks) Validating configuration</font></div><div><font face="courier new, monospace">MainProcess|Thread-397033::ERROR::2012-11-15 12:04:43,110::configNetwork::1155::setupNetworks::(setupNetworks) (24, "Setup attached more than one network to nic em1, some of which aren'</font></div>
<div><font face="courier new, monospace">t vlans")</font></div><div><font face="courier new, monospace">Traceback (most recent call last):</font></div><div><font face="courier new, monospace"> File "/usr/share/vdsm/configNetwork.py", line 1088, in setupNetworks</font></div>
<div><font face="courier new, monospace"> _validateNetworkSetup(dict(networks), dict(bondings))</font></div><div><font face="courier new, monospace"> File "/usr/share/vdsm/configNetwork.py", line 969, in _validateNetworkSetup</font></div>
<div><font face="courier new, monospace"> "Setup attached more than one network to nic %s, some of which aren't vlans"%(nic))</font></div><div><font face="courier new, monospace">ConfigNetworkError: (24, "Setup attached more than one network to nic em1, some of which aren't vlans")</font></div>
<div><font face="courier new, monospace">MainProcess|Thread-397033::ERROR::2012-11-15 12:04:43,112::supervdsmServer::76::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks</font></div><div><font face="courier new, monospace">Traceback (most recent call last):</font></div>
<div><font face="courier new, monospace"> File "/usr/share/vdsm/supervdsmServer.py", line 74, in wrapper</font></div><div><font face="courier new, monospace"> return func(*args, **kwargs)</font></div><div><font face="courier new, monospace"> File "/usr/share/vdsm/supervdsmServer.py", line 150, in setupNetworks</font></div>
<div><font face="courier new, monospace"> return configNetwork.setupNetworks(networks, bondings, **options)</font></div><div><font face="courier new, monospace"> File "/usr/share/vdsm/configNetwork.py", line 1088, in setupNetworks</font></div>
<div><font face="courier new, monospace"> _validateNetworkSetup(dict(networks), dict(bondings))</font></div><div><font face="courier new, monospace"> File "/usr/share/vdsm/configNetwork.py", line 969, in _validateNetworkSetup</font></div>
<div><font face="courier new, monospace"> "Setup attached more than one network to nic %s, some of which aren't vlans"%(nic))</font></div><div><font face="courier new, monospace">ConfigNetworkError: (24, "Setup attached more than one network to nic em1, some of which aren't vlans")</font></div>
<div><font face="courier new, monospace">Thread-397033::ERROR::2012-11-15 12:04:43,112::API::1158::vds::(setupNetworks) Setup attached more than one network to nic em1, some of which aren't vlans</font></div><div><font face="courier new, monospace">Traceback (most recent call last):</font></div>
<div><font face="courier new, monospace"> File "/usr/share/vdsm/API.py", line 1156, in setupNetworks</font></div><div><font face="courier new, monospace"> supervdsm.getProxy().setupNetworks(networks, bondings, options)</font></div>
<div><font face="courier new, monospace"> File "/usr/share/vdsm/supervdsm.py", line 67, in __call__</font></div><div><font face="courier new, monospace"> return callMethod()</font></div><div><font face="courier new, monospace"> File "/usr/share/vdsm/supervdsm.py", line 65, in <lambda></font></div>
<div><font face="courier new, monospace"> **kwargs)</font></div><div><font face="courier new, monospace"> File "<string>", line 2, in setupNetworks</font></div><div><font face="courier new, monospace"> File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod</font></div>
<div><font face="courier new, monospace"> raise convert_to_error(kind, result)</font></div><div><font face="courier new, monospace">ConfigNetworkError: (24, "Setup attached more than one network to nic em1, some of which aren't vlans")</font></div>
<div><font face="courier new, monospace">Thread-397033::DEBUG::2012-11-15 12:04:43,113::BindingXMLRPC::865::vds::(wrapper) return setupNetworks with {'status': {'message': "Setup attached more than one network to nic em1, some of which aren't vlans", 'code': 24}}</font></div>
<div><font face="courier new, monospace">Dummy-3930::DEBUG::2012-11-15 12:04:43,732::__init__::1164::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/88c14568-2860-11e2-8ecf-001a4aa8770c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)</font></div>
<div><font face="courier new, monospace">Dummy-3930::DEBUG::2012-11-15 12:04:43,841::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0565243 s, 18.1 MB/s\n'; <rc> = 0</font></div>
<div><font face="courier new, monospace">VM Channels Listener::DEBUG::2012-11-15 12:04:45,065::vmChannels::60::vds::(_handle_timeouts) Timeout on fileno 317.</font></div><div><font face="courier new, monospace">Dummy-3930::DEBUG::2012-11-15 12:04:45,852::__init__::1164::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/88c14568-2860-11e2-8ecf-001a4aa8770c/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)</font></div>
<div><font face="courier new, monospace">Dummy-3930::DEBUG::2012-11-15 12:04:45,960::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0565389 s, 18.1 MB/s\n'; <rc> = 0</font></div>
</div><div><br></div><div><br></div><div>[A side note: is it wise to keep DEBUG on VDSM logs? If not, is there an easy way to shut it off? That is an awful lot of traffic and might explain why my nodes ran so slow when I was booting off of USB sticks.]</div>
<div><br></div><div>So, all that seems to be consistent with Igor's comments and I'll have to do some fancy dancing to get this setup once I get past the other bug. It would be nice if the GUI relayed the error "<span style="font-family:'courier new',monospace">Setup attached more than one network to nic em1, some of which aren't vlans</span><font face="arial, helvetica, sans-serif">" from VDSM to the user, or some clearer translation, but it sounds like that is on the way based on Moti's comments about the nightly's, so I'll let it drop.</font></div>
<div><br></div><div>This engine and node were built last week and the process went much more smoothly than this first engine having problems. For the problematic engine, which I setup a couple of months ago, the CentOS6 packages had bunch of broken symlinks to jars that were on the system, but not where the symlinks expected them to be. That appears to have been fixed as of least week when I setup this second engine. To fix it on the first, I manually created symlinks where the broken ones were pointed rather than messing with any symlinks created by the packages. </div>
<div><br></div><div>I wonder now if the packages on the bad engine (which are up to date) would have created different symlinks where I did and mine are blocking them from being created. I can rebuild the engine server if needed, but I would appreciate if some one can confirm that this error is consistent with a wrong version of a jar or some similar potential issue. Of course, I would also appreciate if there is a simpler solution. =)</div>