u/s CI job failure on infra issue

[Adam Litke]

On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
>On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
>> Hi,
>>
>> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/ failed on an infra issue.
>> There are more patches that fail like this.
>> Is there indeed an infra issue?
>> If so, what is it?
>> If not, what should be done in the job configuration as to not inform on an infra issue?
>
>All I can tell is that _attachStorageDomain failed with
>
>  AssertionError: error code: 16, message: Unexpected exception
>
>I wish I could get a grip of the vdsm.log during this test. I am not
>sure that
>http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/artifact/vdsm.log
>is the whole thing, but it includes very interesting tracebacks that
>should be better understood.
>
>This is most likely a bug in Vdsm code:
>    Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed
>    Traceback (most recent call last):
>      File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections
>        self._checkConnections()
>      File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections
>        for conId, con in self._conDict.iteritems():
>    RuntimeError: dictionary changed size during iteration
>
>And this one looks like a mom bug:
>    Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error
>    Traceback (most recent call last):
>      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper
>        res = f(*args, **kwargs)
>      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats
>        return api.getStats()
>      File "/usr/share/vdsm/API.py", line 1268, in getStats
>        stats.update(self._cif.mom.getKsmStats())
>      File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
>        stats = self._mom.getStatistics()['host']
>      File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics
>        host_stats = self.threads['host_monitor'].interrogate().statistics[-1]
>    AttributeError: 'NoneType' object has no attribute 'statistics'

This is because 54d16950 was merged in vdsm before mom could support
the named Collectors and Controllers.  Attempting to start MOM with
missing plugins is a fatal error and MOM shuts down.  We can either
revert 54d16950 or wait for the mom part to be merged.  In the later
case, we'll need to ship our own mom rpm with the release since these
changes will not be in the standard Fedora package for awhile.


-- 
Adam Litke