u/s CI job failure on infra issue

[Vered Volansky]

----- Original Message -----
> From: "Dan Kenigsberg" <danken at redhat.com>
> To: "Vered Volansky" <vered at redhat.com>, alitke at redhat.com
> Cc: "infra" <infra at ovirt.org>
> Sent: Tuesday, June 17, 2014 3:35:42 PM
> Subject: Re: u/s CI job failure on infra issue
> 
> On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
> > Hi,
> > 
> > http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/
> > failed on an infra issue.
> > There are more patches that fail like this.
> > Is there indeed an infra issue?
> > If so, what is it?
> > If not, what should be done in the job configuration as to not inform on an
> > infra issue?
> 
> All I can tell is that _attachStorageDomain failed with
> 
>   AssertionError: error code: 16, message: Unexpected exception
> 
> I wish I could get a grip of the vdsm.log during this test. I am not
> sure that
> http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_gerrit/1298/artifact/vdsm.log
> is the whole thing, but it includes very interesting tracebacks that
> should be better understood.
> 
> This is most likely a bug in Vdsm code:
Exactly, this is what it looked like to me, but the infra message is misleading and I was wondering how to remove (assuming that there isn't an infra issue on top the vdsm code obvious issue).

>     Thread-11::ERROR::2014-06-16
>     15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections)
>     Monitoring failed
>     Traceback (most recent call last):
>       File "/usr/share/vdsm/storage/storageServer.py", line 696, in
>       _monitorConnections
>         self._checkConnections()
>       File "/usr/share/vdsm/storage/storageServer.py", line 670, in
>       _checkConnections
>         for conId, con in self._conDict.iteritems():
>     RuntimeError: dictionary changed size during iteration
> 
> And this one looks like a mom bug:
>     Thread-113::ERROR::2014-06-16
>     15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error
>     Traceback (most recent call last):
>       File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper
>         res = f(*args, **kwargs)
>       File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats
>         return api.getStats()
>       File "/usr/share/vdsm/API.py", line 1268, in getStats
>         stats.update(self._cif.mom.getKsmStats())
>       File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
>         stats = self._mom.getStatistics()['host']
>       File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in
>       getStatistics
>         host_stats =
>         self.threads['host_monitor'].interrogate().statistics[-1]
>     AttributeError: 'NoneType' object has no attribute 'statistics'
>