Re: u/s CI job failure on infra issue

On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote: > Hi, > > http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs... failed on an infra issue. > There are more patches that fail like this. > Is there indeed an infra issue? > If so, what is it? > If not, what should be done in the job configuration as to not inform on an infra issue? All I can tell is that _attachStorageDomain failed with AssertionError: error code: 16, message: Unexpected exception I wish I could get a grip of the vdsm.log during this test. I am not sure that http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs... is the whole thing, but it includes very interesting tracebacks that should be better understood. This is most likely a bug in Vdsm code: Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed Traceback (most recent call last): File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections self._checkConnections() File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections for conId, con in self._conDict.iteritems(): RuntimeError: dictionary changed size during iteration And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'