oVirt 4.2.5.1-1.el7 JSON-RPC statistics error
I've got one physical host in a 3 host CentOS7.5 cluster that reports the following error several times a day VDSM node3.example.com command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '<Fault 1: "<type \'exceptions.AttributeError\'>:\'NoneType\' object has no attribute \'statistics\'">'} Any ideas what the problem might be?
On 07/17/2018 07:30 AM, Maton, Brett wrote:
I've got one physical host in a 3 host CentOS7.5 cluster that reports the following error several times a day
VDSM node3.example.com <http://node3.example.com> command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '<Fault 1: "<type \'exceptions.AttributeError\'>:\'NoneType\' object has no attribute \'statistics\'">'}
Any ideas what the problem might be?
Hi, I can make a wild guess[1], but let's try educated guesses first: could you please share the Vdsm logs around the time on which you see this error? Thanks, +++ [1] It seems to me that a NIC card was removed from a VM during a statistic reporting cycle, or somehow failed to report stats. But really, is a wild guess.Let's look at the logs first. -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Thanks Francesco, Log attached. On 17 July 2018 at 13:12, Francesco Romani <fromani@redhat.com> wrote:
On 07/17/2018 07:30 AM, Maton, Brett wrote:
I've got one physical host in a 3 host CentOS7.5 cluster that reports the following error several times a day
VDSM node3.example.com command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '<Fault 1: "<type \'exceptions.AttributeError\'>:\'NoneType\' object has no attribute \'statistics\'">'}
Any ideas what the problem might be?
Hi,
I can make a wild guess[1], but let's try educated guesses first: could you please share the Vdsm logs around the time on which you see this error?
Thanks,
+++
[1] It seems to me that a NIC card was removed from a VM during a statistic reporting cycle, or somehow failed to report stats. But really, is a wild guess.Let's look at the logs first.
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ message/JSRLKAZSSPXGUBDE7CXRQDXZTQJBY5YY/
On 07/18/2018 07:24 AM, Maton, Brett wrote:
Thanks Francesco,
Log attached.
Interestings, it seems the fault comes from MOM: 2018-07-18 06:07:22,836+0100 INFO (jsonrpc/3) [api.host] FINISH getStats error=<Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'statistics'"> from=::ffff:10.11.30.253,58840 (api:50) 2018-07-18 06:07:22,836+0100 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1402, in getStats sampling.host_samples.stats())} File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 75, in get_stats ret.update(cif.mom.getKsmStats()) File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 75, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1306, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/xmlrpclib.py", line 1482, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 794, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'statistics'"> 2018-07-18 06:07:22,836+0100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats failed (error -32603) in 0.01 seconds (__init__:573) 2018-07-18 06:07:22,909+0100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,60058, vmId=a1af259b-7917-49ac-8bcc-eb4cc302e723 (api:46) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'a1af259b-7917-49ac-8bcc-eb4cc302e723'} (api:127) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'a1af259b-7917-49ac-8bcc-eb4cc302e723'}", 'code': 1}} from=::1,60058, vmId=a1af259b-7917-49ac-8bcc-eb4cc302e723 (api:52) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:573) 2018-07-18 06:07:23,043+0100 INFO (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:110) It could be a (benign) race on shutdown, but better continue the investigation. Could you please also chare the mom.log corresponding to the timeframe you already provided? Thanks, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Sure no problem, mom log attached. On 18 July 2018 at 12:36, Francesco Romani <fromani@redhat.com> wrote:
On 07/18/2018 07:24 AM, Maton, Brett wrote:
Thanks Francesco,
Log attached.
Interestings, it seems the fault comes from MOM:
2018-07-18 06:07:22,836+0100 INFO (jsonrpc/3) [api.host] FINISH getStats error=<Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'statistics'"> from=::ffff:10.11.30.253,58840 (api:50) 2018-07-18 06:07:22,836+0100 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1402, in getStats sampling.host_samples.stats())} File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 75, in get_stats ret.update(cif.mom.getKsmStats()) File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 75, in getKsmStats stats = self._mom.getStatistics()['host'] File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1306, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/xmlrpclib.py", line 1482, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 794, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'statistics'"> 2018-07-18 06:07:22,836+0100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats failed (error -32603) in 0.01 seconds (__init__:573) 2018-07-18 06:07:22,909+0100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,60058, vmId=a1af259b-7917-49ac-8bcc-eb4cc302e723 (api:46) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'a1af259b-7917-49ac-8bcc-eb4cc302e723'} (api:127) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'a1af259b-7917-49ac-8bcc-eb4cc302e723'}", 'code': 1}} from=::1,60058, vmId=a1af259b-7917-49ac-8bcc-eb4cc302e723 (api:52) 2018-07-18 06:07:22,912+0100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:573) 2018-07-18 06:07:23,043+0100 INFO (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:110)
It could be a (benign) race on shutdown, but better continue the investigation. Could you please also chare the mom.log corresponding to the timeframe you already provided?
Thanks,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Thanks! On 07/18/2018 02:11 PM, Maton, Brett wrote:
Sure no problem, mom log attached.
It seems to me that MOM is restarting *very* often: 2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting And I totally not expect this. Martin, Andrej, is this the expected behaviour? Bests, -- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
FWIW: This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz Network is bonded fail-over. Regards, Brett On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
Thanks!
On 07/18/2018 02:11 PM, Maton, Brett wrote:
Sure no problem, mom log attached.
It seems to me that MOM is restarting *very* often:
2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting
And I totally not expect this. Martin, Andrej, is this the expected behaviour?
Bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Hi, Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version. Updating MOM to version 0.5.12 should fix it. Regards, Andrej On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
FWIW:
This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Network is bonded fail-over.
Regards, Brett
On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
Thanks!
On 07/18/2018 02:11 PM, Maton, Brett wrote:
Sure no problem, mom log attached.
It seems to me that MOM is restarting *very* often:
2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting
And I totally not expect this. Martin, Andrej, is this the expected behaviour?
Bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
I just checked the mom version, it's already at 0.5.12 # rpm -qa mom mom-0.5.12-1.el7.centos.noarch All the physical hosts are fully updated /patched. Regards, Brett On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version.
Updating MOM to version 0.5.12 should fix it.
Regards, Andrej
On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
FWIW:
This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Network is bonded fail-over.
Regards, Brett
On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
Thanks!
On 07/18/2018 02:11 PM, Maton, Brett wrote:
Sure no problem, mom log attached.
It seems to me that MOM is restarting *very* often:
2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting
And I totally not expect this. Martin, Andrej, is this the expected behaviour?
Bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Maybe the python module is missing. Can you check if this file exists? /usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py Andrej On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
I just checked the mom version, it's already at 0.5.12
# rpm -qa mom mom-0.5.12-1.el7.centos.noarch
All the physical hosts are fully updated /patched.
Regards, Brett
On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version.
Updating MOM to version 0.5.12 should fix it.
Regards, Andrej
On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
FWIW:
This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Network is bonded fail-over.
Regards, Brett
On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
Thanks!
On 07/18/2018 02:11 PM, Maton, Brett wrote:
Sure no problem, mom log attached.
It seems to me that MOM is restarting *very* often:
2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting
And I totally not expect this. Martin, Andrej, is this the expected behaviour?
Bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Bingo How could that file not be installed/deployed ? Should I simply copy it from one of the other hosts to make the message go away ? On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir@redhat.com> wrote:
Maybe the python module is missing. Can you check if this file exists?
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/ vdsmjsonrpcclientInterface.py
Andrej
On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
I just checked the mom version, it's already at 0.5.12
# rpm -qa mom mom-0.5.12-1.el7.centos.noarch
All the physical hosts are fully updated /patched.
Regards, Brett
On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version.
Updating MOM to version 0.5.12 should fix it.
Regards, Andrej
On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
FWIW:
This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Network is bonded fail-over.
Regards, Brett
On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
Thanks!
On 07/18/2018 02:11 PM, Maton, Brett wrote:
Sure no problem, mom log attached.
It seems to me that MOM is restarting *very* often:
2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting
And I totally not expect this. Martin, Andrej, is this the expected behaviour?
Bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Yes, copying it from another host with mom version 0.5.12 is enough. I'm not sure why it was not installed. The file is in the rpm. On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb@ltresources.co.uk> wrote:
Bingo
How could that file not be installed/deployed ?
Should I simply copy it from one of the other hosts to make the message go away ?
On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir@redhat.com> wrote:
Maybe the python module is missing. Can you check if this file exists?
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py
Andrej
On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
I just checked the mom version, it's already at 0.5.12
# rpm -qa mom mom-0.5.12-1.el7.centos.noarch
All the physical hosts are fully updated /patched.
Regards, Brett
On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version.
Updating MOM to version 0.5.12 should fix it.
Regards, Andrej
On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
FWIW:
This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Network is bonded fail-over.
Regards, Brett
On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
Thanks!
On 07/18/2018 02:11 PM, Maton, Brett wrote:
> Sure no problem, mom log attached. >
It seems to me that MOM is restarting *very* often:
2018-07-18 06:07:04,909 - mom - INFO - MOM starting 2018-07-18 06:07:20,407 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting 2018-07-18 06:07:30,658 - mom - INFO - MOM starting
And I totally not expect this. Martin, Andrej, is this the expected behaviour?
Bests,
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh
Thanks, Cluster is all installed from pre-release, maybe I managed to get an iffy rpm.... On 18 July 2018 at 15:50, Andrej Krejcir <akrejcir@redhat.com> wrote:
Yes, copying it from another host with mom version 0.5.12 is enough.
I'm not sure why it was not installed. The file is in the rpm.
On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb@ltresources.co.uk> wrote:
Bingo
How could that file not be installed/deployed ?
Should I simply copy it from one of the other hosts to make the message go away ?
On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir@redhat.com> wrote:
Maybe the python module is missing. Can you check if this file exists?
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/ vdsmjsonrpcclientInterface.py
Andrej
On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
I just checked the mom version, it's already at 0.5.12
# rpm -qa mom mom-0.5.12-1.el7.centos.noarch
All the physical hosts are fully updated /patched.
Regards, Brett
On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version.
Updating MOM to version 0.5.12 should fix it.
Regards, Andrej
On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
FWIW:
This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Network is bonded fail-over.
Regards, Brett
On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> wrote:
> Thanks! > > > On 07/18/2018 02:11 PM, Maton, Brett wrote: > >> Sure no problem, mom log attached. >> > > It seems to me that MOM is restarting *very* often: > > 2018-07-18 06:07:04,909 - mom - INFO - MOM starting > 2018-07-18 06:07:20,407 - mom - INFO - MOM starting > 2018-07-18 06:07:30,658 - mom - INFO - MOM starting > 2018-07-18 06:07:30,658 - mom - INFO - MOM starting > > And I totally not expect this. Martin, Andrej, is this the expected > behaviour? > > > Bests, > > > > -- > Francesco Romani > Senior SW Eng., Virtualization R&D > Red Hat > IRC: fromani github: @fromanirh > >
I upgraded my test cluster to 4.2.5.2-1 last night (hosts rebooted after update) and I've started to see the error again. As before it seems to be limited to the one host, I've checked that /usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py exists, which it does but there are no matching .pyc / .pyo files which suggests that it's not been used? File permissions and SELinux contexts are the same on all files in the HypervisorInterfaces directory. 0644 root root system_u:object_r:lib_t:s0 vdsm / mom log from time period attached. On 18 July 2018 at 16:10, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks,
Cluster is all installed from pre-release, maybe I managed to get an iffy rpm....
On 18 July 2018 at 15:50, Andrej Krejcir <akrejcir@redhat.com> wrote:
Yes, copying it from another host with mom version 0.5.12 is enough.
I'm not sure why it was not installed. The file is in the rpm.
On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb@ltresources.co.uk> wrote:
Bingo
How could that file not be installed/deployed ?
Should I simply copy it from one of the other hosts to make the message go away ?
On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir@redhat.com> wrote:
Maybe the python module is missing. Can you check if this file exists?
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vd smjsonrpcclientInterface.py
Andrej
On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
I just checked the mom version, it's already at 0.5.12
# rpm -qa mom mom-0.5.12-1.el7.centos.noarch
All the physical hosts are fully updated /patched.
Regards, Brett
On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Looking at MOM logs, the problem seems to be that MOM is configured to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot find this module, probably because it is an older version.
Updating MOM to version 0.5.12 should fix it.
Regards, Andrej
On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> wrote:
> FWIW: > > This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) > Xeon(R) CPU E3-1220 V2 @ 3.10GHz > > Network is bonded fail-over. > > Regards, > Brett > > On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> > wrote: > >> Thanks! >> >> >> On 07/18/2018 02:11 PM, Maton, Brett wrote: >> >>> Sure no problem, mom log attached. >>> >> >> It seems to me that MOM is restarting *very* often: >> >> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >> >> And I totally not expect this. Martin, Andrej, is this the expected >> behaviour? >> >> >> Bests, >> >> >> >> -- >> Francesco Romani >> Senior SW Eng., Virtualization R&D >> Red Hat >> IRC: fromani github: @fromanirh >> >> >
I think the upgrade may not have affected the mom package, because no new mom rpm version was released for a few months. I'm not sure why the import fails even if the file is there. As a workaround, mom can be configured to use the older (deprecated) json rpc interface to communicate with vdsm. There is no difference in functionality. This can be done by editing /etc/vdsm/mom.conf. Set the 'hypervisor-interface' config value to 'vdsmjsonrpcbulk' insted of 'vdsmjsonrpcclient'. On Wed, 25 Jul 2018 at 08:56, Maton, Brett <matonb@ltresources.co.uk> wrote:
I upgraded my test cluster to 4.2.5.2-1 last night (hosts rebooted after update) and I've started to see the error again.
As before it seems to be limited to the one host, I've checked that
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py
exists, which it does but there are no matching .pyc / .pyo files which suggests that it's not been used?
File permissions and SELinux contexts are the same on all files in the HypervisorInterfaces directory.
0644 root root system_u:object_r:lib_t:s0
vdsm / mom log from time period attached.
On 18 July 2018 at 16:10, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks,
Cluster is all installed from pre-release, maybe I managed to get an iffy rpm....
On 18 July 2018 at 15:50, Andrej Krejcir <akrejcir@redhat.com> wrote:
Yes, copying it from another host with mom version 0.5.12 is enough.
I'm not sure why it was not installed. The file is in the rpm.
On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb@ltresources.co.uk> wrote:
Bingo
How could that file not be installed/deployed ?
Should I simply copy it from one of the other hosts to make the message go away ?
On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir@redhat.com> wrote:
Maybe the python module is missing. Can you check if this file exists?
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py
Andrej
On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
I just checked the mom version, it's already at 0.5.12
# rpm -qa mom mom-0.5.12-1.el7.centos.noarch
All the physical hosts are fully updated /patched.
Regards, Brett
On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> wrote:
> Hi, > > Looking at MOM logs, the problem seems to be that MOM is configured > to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it > cannot find this module, probably because it is an older version. > > Updating MOM to version 0.5.12 should fix it. > > > Regards, > Andrej > > On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb@ltresources.co.uk> > wrote: > >> FWIW: >> >> This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) >> Xeon(R) CPU E3-1220 V2 @ 3.10GHz >> >> Network is bonded fail-over. >> >> Regards, >> Brett >> >> On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> >> wrote: >> >>> Thanks! >>> >>> >>> On 07/18/2018 02:11 PM, Maton, Brett wrote: >>> >>>> Sure no problem, mom log attached. >>>> >>> >>> It seems to me that MOM is restarting *very* often: >>> >>> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >>> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>> >>> And I totally not expect this. Martin, Andrej, is this the >>> expected behaviour? >>> >>> >>> Bests, >>> >>> >>> >>> -- >>> Francesco Romani >>> Senior SW Eng., Virtualization R&D >>> Red Hat >>> IRC: fromani github: @fromanirh >>> >>> >>
I rebooted the server that was failing last week and haven't seen the error since. On 25 July 2018 at 11:48, Andrej Krejcir <akrejcir@redhat.com> wrote:
I think the upgrade may not have affected the mom package, because no new mom rpm version was released for a few months.
I'm not sure why the import fails even if the file is there.
As a workaround, mom can be configured to use the older (deprecated) json rpc interface to communicate with vdsm. There is no difference in functionality. This can be done by editing /etc/vdsm/mom.conf. Set the 'hypervisor-interface' config value to 'vdsmjsonrpcbulk' insted of 'vdsmjsonrpcclient'.
On Wed, 25 Jul 2018 at 08:56, Maton, Brett <matonb@ltresources.co.uk> wrote:
I upgraded my test cluster to 4.2.5.2-1 last night (hosts rebooted after update) and I've started to see the error again.
As before it seems to be limited to the one host, I've checked that
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/ vdsmjsonrpcclientInterface.py
exists, which it does but there are no matching .pyc / .pyo files which suggests that it's not been used?
File permissions and SELinux contexts are the same on all files in the HypervisorInterfaces directory.
0644 root root system_u:object_r:lib_t:s0
vdsm / mom log from time period attached.
On 18 July 2018 at 16:10, Maton, Brett <matonb@ltresources.co.uk> wrote:
Thanks,
Cluster is all installed from pre-release, maybe I managed to get an iffy rpm....
On 18 July 2018 at 15:50, Andrej Krejcir <akrejcir@redhat.com> wrote:
Yes, copying it from another host with mom version 0.5.12 is enough.
I'm not sure why it was not installed. The file is in the rpm.
On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb@ltresources.co.uk> wrote:
Bingo
How could that file not be installed/deployed ?
Should I simply copy it from one of the other hosts to make the message go away ?
On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir@redhat.com> wrote:
Maybe the python module is missing. Can you check if this file exists?
/usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/ vdsmjsonrpcclientInterface.py
Andrej
On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb@ltresources.co.uk> wrote:
> I just checked the mom version, it's already at 0.5.12 > > # rpm -qa mom > mom-0.5.12-1.el7.centos.noarch > > All the physical hosts are fully updated /patched. > > Regards, > Brett > > On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir@redhat.com> > wrote: > >> Hi, >> >> Looking at MOM logs, the problem seems to be that MOM is configured >> to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it >> cannot find this module, probably because it is an older version. >> >> Updating MOM to version 0.5.12 should fix it. >> >> >> Regards, >> Andrej >> >> On Wed, 18 Jul 2018 at 14:41, Maton, Brett < >> matonb@ltresources.co.uk> wrote: >> >>> FWIW: >>> >>> This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) >>> Xeon(R) CPU E3-1220 V2 @ 3.10GHz >>> >>> Network is bonded fail-over. >>> >>> Regards, >>> Brett >>> >>> On 18 July 2018 at 13:15, Francesco Romani <fromani@redhat.com> >>> wrote: >>> >>>> Thanks! >>>> >>>> >>>> On 07/18/2018 02:11 PM, Maton, Brett wrote: >>>> >>>>> Sure no problem, mom log attached. >>>>> >>>> >>>> It seems to me that MOM is restarting *very* often: >>>> >>>> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >>>> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>> >>>> And I totally not expect this. Martin, Andrej, is this the >>>> expected behaviour? >>>> >>>> >>>> Bests, >>>> >>>> >>>> >>>> -- >>>> Francesco Romani >>>> Senior SW Eng., Virtualization R&D >>>> Red Hat >>>> IRC: fromani github: @fromanirh >>>> >>>> >>> >
participants (3)
-
Andrej Krejcir -
Francesco Romani -
Maton, Brett