Hi, http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... failed on an infra issue. There are more patches that fail like this. Is there indeed an infra issue? If so, what is it? If not, what should be done in the job configuration as to not inform on an infra issue? Thanks, Vered
On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
Hi,
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... failed on an infra issue. There are more patches that fail like this. Is there indeed an infra issue? If so, what is it? If not, what should be done in the job configuration as to not inform on an infra issue?
All I can tell is that _attachStorageDomain failed with AssertionError: error code: 16, message: Unexpected exception I wish I could get a grip of the vdsm.log during this test. I am not sure that http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... is the whole thing, but it includes very interesting tracebacks that should be better understood. This is most likely a bug in Vdsm code: Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed Traceback (most recent call last): File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections self._checkConnections() File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections for conId, con in self._conDict.iteritems(): RuntimeError: dictionary changed size during iteration And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Vered Volansky" <vered@redhat.com>, alitke@redhat.com Cc: "infra" <infra@ovirt.org> Sent: Tuesday, June 17, 2014 3:35:42 PM Subject: Re: u/s CI job failure on infra issue
On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
Hi,
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... failed on an infra issue. There are more patches that fail like this. Is there indeed an infra issue? If so, what is it? If not, what should be done in the job configuration as to not inform on an infra issue?
All I can tell is that _attachStorageDomain failed with
AssertionError: error code: 16, message: Unexpected exception
I wish I could get a grip of the vdsm.log during this test. I am not sure that http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... is the whole thing, but it includes very interesting tracebacks that should be better understood.
This is most likely a bug in Vdsm code: Exactly, this is what it looked like to me, but the infra message is misleading and I was wondering how to remove (assuming that there isn't an infra issue on top the vdsm code obvious issue).
Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed Traceback (most recent call last): File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections self._checkConnections() File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections for conId, con in self._conDict.iteritems(): RuntimeError: dictionary changed size during iteration
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
Hi,
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... failed on an infra issue. There are more patches that fail like this. Is there indeed an infra issue? If so, what is it? If not, what should be done in the job configuration as to not inform on an infra issue?
All I can tell is that _attachStorageDomain failed with
AssertionError: error code: 16, message: Unexpected exception
I wish I could get a grip of the vdsm.log during this test. I am not sure that http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... is the whole thing, but it includes very interesting tracebacks that should be better understood.
This is most likely a bug in Vdsm code: Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed Traceback (most recent call last): File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections self._checkConnections() File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections for conId, con in self._conDict.iteritems(): RuntimeError: dictionary changed size during iteration
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
This is because 54d16950 was merged in vdsm before mom could support the named Collectors and Controllers. Attempting to start MOM with missing plugins is a fatal error and MOM shuts down. We can either revert 54d16950 or wait for the mom part to be merged. In the later case, we'll need to ship our own mom rpm with the release since these changes will not be in the standard Fedora package for awhile. -- Adam Litke
On Tue, Jun 17, 2014 at 09:19:09AM -0400, Adam Litke wrote:
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
This is because 54d16950 was merged in vdsm before mom could support the named Collectors and Controllers. Attempting to start MOM with missing plugins is a fatal error and MOM shuts down. We can either revert 54d16950 or wait for the mom part to be merged. In the later case, we'll need to ship our own mom rpm with the release since these changes will not be in the standard Fedora package for awhile.
I believe that CPU tuning was deemed a required 3.5 feature, so reverting the patch should be done as a last resort. In any case, 54d16950 should have included a strict dependency of a mom version. What is the version that provides named Collectors? Would this version be available for Jenkins (and humans) to test and install? Regrads, Dan.
On 17/06/14 15:14 +0100, Dan Kenigsberg wrote:
On Tue, Jun 17, 2014 at 09:19:09AM -0400, Adam Litke wrote:
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
This is because 54d16950 was merged in vdsm before mom could support the named Collectors and Controllers. Attempting to start MOM with missing plugins is a fatal error and MOM shuts down. We can either revert 54d16950 or wait for the mom part to be merged. In the later case, we'll need to ship our own mom rpm with the release since these changes will not be in the standard Fedora package for awhile.
I believe that CPU tuning was deemed a required 3.5 feature, so reverting the patch should be done as a last resort.
In any case, 54d16950 should have included a strict dependency of a mom version. What is the version that provides named Collectors? Would this version be available for Jenkins (and humans) to test and install?
It's not even merged yet. Kobi is still working on the patches.
Regrads, Dan.
-- Adam Litke
----- Original Message -----
From: "Adam Litke" <alitke@redhat.com> To: "Dan Kenigsberg" <danken@redhat.com> Cc: kobi@redhat.com, "Vered Volansky" <vered@redhat.com>, "infra" <infra@ovirt.org>, "Doron Fediuck" <dfediuck@redhat.com>, "Martin Sivak" <msivak@redhat.com> Sent: Tuesday, June 17, 2014 5:20:34 PM Subject: Re: u/s CI job failure on infra issue
On 17/06/14 15:14 +0100, Dan Kenigsberg wrote:
On Tue, Jun 17, 2014 at 09:19:09AM -0400, Adam Litke wrote:
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
This is because 54d16950 was merged in vdsm before mom could support the named Collectors and Controllers. Attempting to start MOM with missing plugins is a fatal error and MOM shuts down. We can either revert 54d16950 or wait for the mom part to be merged. In the later case, we'll need to ship our own mom rpm with the release since these changes will not be in the standard Fedora package for awhile.
I believe that CPU tuning was deemed a required 3.5 feature, so reverting the patch should be done as a last resort.
In any case, 54d16950 should have included a strict dependency of a mom version. What is the version that provides named Collectors? Would this version be available for Jenkins (and humans) to test and install?
It's not even merged yet. Kobi is still working on the patches.
Actually CPU parts are merged into vdsm and partially in the engine as well. Indeed as Dan mentioned this is a gating item which we need to conclude for 3.5 so reverting should be avoided if there's a way to handle it. There was a patch initially merged to mom and reverted due to vdsm changes. So I'm asking to stop the reverting sequence and make sure we have all current patches in place to get cpu working. Martin Sivak is assisting Kobi so we should have a resolution soon (if not already posted).
Regrads, Dan.
-- Adam Litke
----- Original Message -----
From: "Doron Fediuck" <dfediuck@redhat.com> To: "Adam Litke" <alitke@redhat.com> Cc: "Dan Kenigsberg" <danken@redhat.com>, kobi@redhat.com, "Vered Volansky" <vered@redhat.com>, "infra" <infra@ovirt.org>, "Martin Sivak" <msivak@redhat.com> Sent: Tuesday, June 17, 2014 5:28:55 PM Subject: Re: u/s CI job failure on infra issue
----- Original Message -----
From: "Adam Litke" <alitke@redhat.com> To: "Dan Kenigsberg" <danken@redhat.com> Cc: kobi@redhat.com, "Vered Volansky" <vered@redhat.com>, "infra" <infra@ovirt.org>, "Doron Fediuck" <dfediuck@redhat.com>, "Martin Sivak" <msivak@redhat.com> Sent: Tuesday, June 17, 2014 5:20:34 PM Subject: Re: u/s CI job failure on infra issue
On 17/06/14 15:14 +0100, Dan Kenigsberg wrote:
On Tue, Jun 17, 2014 at 09:19:09AM -0400, Adam Litke wrote:
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
This is because 54d16950 was merged in vdsm before mom could support the named Collectors and Controllers. Attempting to start MOM with missing plugins is a fatal error and MOM shuts down. We can either revert 54d16950 or wait for the mom part to be merged. In the later case, we'll need to ship our own mom rpm with the release since these changes will not be in the standard Fedora package for awhile.
I believe that CPU tuning was deemed a required 3.5 feature, so reverting the patch should be done as a last resort.
In any case, 54d16950 should have included a strict dependency of a mom version. What is the version that provides named Collectors? Would this version be available for Jenkins (and humans) to test and install?
It's not even merged yet. Kobi is still working on the patches.
Actually CPU parts are merged into vdsm and partially in the engine as well. Indeed as Dan mentioned this is a gating item which we need to conclude for 3.5 so reverting should be avoided if there's a way to handle it. There was a patch initially merged to mom and reverted due to vdsm changes.
So I'm asking to stop the reverting sequence and make sure we have all current patches in place to get cpu working. Martin Sivak is assisting Kobi so we should have a resolution soon (if not already posted).
Got a fresh patch of mom submitted this morning. this includes: mom make fix vdsm api fix mom standalone libvirt api fix verified as well. Adam - can you please review it
Regrads, Dan.
-- Adam Litke
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
Hi,
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... failed on an infra issue. There are more patches that fail like this. Is there indeed an infra issue? If so, what is it? If not, what should be done in the job configuration as to not inform on an infra issue?
All I can tell is that _attachStorageDomain failed with
AssertionError: error code: 16, message: Unexpected exception
I wish I could get a grip of the vdsm.log during this test. I am not sure that http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... is the whole thing, but it includes very interesting tracebacks that should be better understood.
This is most likely a bug in Vdsm code: Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed Traceback (most recent call last): File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections self._checkConnections() File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections for conId, con in self._conDict.iteritems(): RuntimeError: dictionary changed size during iteration
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
Until this is fixed, the following command can be used to fix your source so the problem will not appear: git show 54d16950 -- vdsm/mom.conf.in | patch -p1 -R -- Adam Litke
----- Original Message -----
From: "Adam Litke" <alitke@redhat.com> To: "Dan Kenigsberg" <danken@redhat.com> Cc: "Vered Volansky" <vered@redhat.com>, "infra" <infra@ovirt.org> Sent: Tuesday, June 17, 2014 7:08:42 PM Subject: Re: u/s CI job failure on infra issue
On 17/06/14 13:35 +0100, Dan Kenigsberg wrote:
On Tue, Jun 17, 2014 at 03:02:07AM -0400, Vered Volansky wrote:
Hi,
http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... failed on an infra issue. There are more patches that fail like this. Is there indeed an infra issue? If so, what is it? If not, what should be done in the job configuration as to not inform on an infra issue?
All I can tell is that _attachStorageDomain failed with
AssertionError: error code: 16, message: Unexpected exception
I wish I could get a grip of the vdsm.log during this test. I am not sure that http://jenkins.ovirt.org/job/vdsm_master_storage_functional_tests_localfs_ge... is the whole thing, but it includes very interesting tracebacks that should be better understood.
This is most likely a bug in Vdsm code: Thread-11::ERROR::2014-06-16 15:22:42,157::storageServer::704::Storage.ConnectionMonitor::(_monitorConnections) Monitoring failed Traceback (most recent call last): File "/usr/share/vdsm/storage/storageServer.py", line 696, in _monitorConnections self._checkConnections() File "/usr/share/vdsm/storage/storageServer.py", line 670, in _checkConnections for conId, con in self._conDict.iteritems(): RuntimeError: dictionary changed size during iteration
And this one looks like a mom bug: Thread-113::ERROR::2014-06-16 15:22:34,115::BindingXMLRPC::1126::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 1110, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/rpc/BindingXMLRPC.py", line 455, in getStats return api.getStats() File "/usr/share/vdsm/API.py", line 1268, in getStats stats.update(self._cif.mom.getKsmStats()) File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib/python2.7/site-packages/mom/MOMFuncs.py", line 74, in getStatistics host_stats = self.threads['host_monitor'].interrogate().statistics[-1] AttributeError: 'NoneType' object has no attribute 'statistics'
Until this is fixed, the following command can be used to fix your source so the problem will not appear:
git show 54d16950 -- vdsm/mom.conf.in | patch -p1 -R
I'll try that. Thanks, Vered
-- Adam Litke _______________________________________________ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
participants (5)
-
Adam Litke -
Dan Kenigsberg -
Doron Fediuck -
Kobi Ianko -
Vered Volansky