u/s CI job failure on infra issue

[Adam Litke]

On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
>On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
>> Hi,
>>
>> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/ failed on an infra issue.
>> There are more patches that fail like this.
>> Is there indeed an infra issue?
>> If so, what is it?
>> If not, what should be done in the job configuration as to not inform on an infra issue?
>
>All I can tell is that _attachStorageDomain failed with
>
>  AssertionError: error code: 16, message: Unexpected exception
>
>I wish I could get a grip of the vdsm.log during this test. I am not
>sure that
>http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/artifact/vdsm.log
>is the whole thing, but it includes very interesting tracebacks that
>should be better understood.
>
>This is most likely a bug in Vdsm code:
>    Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed
>    Traceback (most recent call last):
>      File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections
>        self._checkConnections()
>      File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections
>        for conId, con in self._conDict.iteritems():
>    RuntimeError: dictionary changed size during iteration
>
>And this one looks like a mom bug:
>    Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error
>    Traceback (most recent call last):
>      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper
>        res = f(*args, **kwargs)
>      File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats
>        return api.getStats()
>      File "/usr/share/vdsm/API.py", line 1268, in getStats
>        stats.update(self._cif.mom.getKsmStats())
>      File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
>        stats = self._mom.getStatistics()['host']
>      File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics
>        host_stats = self.threads['host_monitor'].interrogate().statistics[-1]
>    AttributeError: 'NoneType' object has no attribute 'statistics'

Until this is fixed, the following command can be used to fix your
source so the problem will not appear:

git show 54d16950 -- vdsm/mom.conf.in | patch -p1 -R

-- 
Adam Litke