u/s CI job failure on infra issue

[Dan Kenigsberg]

On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
> Hi,
> 
> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/ failed on an infra issue.
> There are more patches that fail like this.
> Is there indeed an infra issue?
> If so, what is it?
> If not, what should be done in the job configuration as to not inform on an infra issue?

All I can tell is that _attachStorageDomain failed with

  AssertionError: error code: 16, message: Unexpected exception

I wish I could get a grip of the vdsm.log during this test. I am not
sure that
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/artifact/vdsm.log
is the whole thing, but it includes very interesting tracebacks that
should be better understood.

This is most likely a bug in Vdsm code:
    Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed
    Traceback (most recent call last):
      File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections
        self._checkConnections()
      File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections
        for conId, con in self._conDict.iteritems():
    RuntimeError: dictionary changed size during iteration

And this one looks like a mom bug:
    Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error
    Traceback (most recent call last):
      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper
        res = f(*args, **kwargs)
      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats
        return api.getStats()
      File "/usr/share/vdsm/API.py", line 1268, in getStats
        stats.update(self._cif.mom.getKsmStats())
      File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
        stats = self._mom.getStatistics()['host']
      File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics
        host_stats = self.threads['host_monitor'].interrogate().statistics[-1]
    AttributeError: 'NoneType' object has no attribute 'statistics'