I've got one physical host in a 3 host CentOS7.5 cluster that reports the following error several times a day VDSM node3.example.com <http://node3.example.com> command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '<Fault 1: "<type \'exceptions.AttributeError\'>:\'NoneType\' object has no attribute \'statistics\'">'} Any ideas what the problem might be?

Thanks Francesco,   Log attached.

Sure no problem, mom log attached.

FWIW: This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz Network is bonded fail-over. Regards, Brett On 18 July 2018 at 13:15, Francesco Romani <fromani(a)redhat.com&gt; wrote: > Thanks! > > > On 07/18/2018 02:11 PM, Maton, Brett wrote: > >> Sure no problem, mom log attached. >> > > It seems to me that MOM is restarting *very* often: > > 2018-07-18 06:07:04,909 - mom - INFO - MOM starting > 2018-07-18 06:07:20,407 - mom - INFO - MOM starting > 2018-07-18 06:07:30,658 - mom - INFO - MOM starting > 2018-07-18 06:07:30,658 - mom - INFO - MOM starting > > And I totally not expect this. Martin, Andrej, is this the expected > behaviour? > > > Bests, > > > > -- > Francesco Romani > Senior SW Eng., Virtualization R&D > Red Hat > IRC: fromani github: @fromanirh > >

I just checked the mom version, it's already at 0.5.12 # rpm -qa mom mom-0.5.12-1.el7.centos.noarch All the physical hosts are fully updated /patched. Regards, Brett On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: > Hi, > > Looking at MOM logs, the problem seems to be that MOM is configured to > use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot > find this module, probably because it is an older version. > > Updating MOM to version 0.5.12 should fix it. > > > Regards, > Andrej > > On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb(a)ltresources.co.uk&gt; > wrote: > >> FWIW: >> >> This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) Xeon(R) >> CPU E3-1220 V2 @ 3.10GHz >> >> Network is bonded fail-over. >> >> Regards, >> Brett >> >> On 18 July 2018 at 13:15, Francesco Romani <fromani(a)redhat.com&gt; wrote: >> >>> Thanks! >>> >>> >>> On 07/18/2018 02:11 PM, Maton, Brett wrote: >>> >>>> Sure no problem, mom log attached. >>>> >>> >>> It seems to me that MOM is restarting *very* often: >>> >>> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >>> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>> >>> And I totally not expect this. Martin, Andrej, is this the expected >>> behaviour? >>> >>> >>> Bests, >>> >>> >>> >>> -- >>> Francesco Romani >>> Senior SW Eng., Virtualization R&D >>> Red Hat >>> IRC: fromani github: @fromanirh >>> >>> >>

Bingo How could that file not be installed/deployed ? Should I simply copy it from one of the other hosts to make the message go away ? On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: > Maybe the python module is missing. Can you check if this file exists? > > > /usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vdsmjsonrpcclientInterface.py > > > Andrej > > On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb(a)ltresources.co.uk&gt; > wrote: > >> I just checked the mom version, it's already at 0.5.12 >> >> # rpm -qa mom >> mom-0.5.12-1.el7.centos.noarch >> >> All the physical hosts are fully updated /patched. >> >> Regards, >> Brett >> >> On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: >> >>> Hi, >>> >>> Looking at MOM logs, the problem seems to be that MOM is configured to >>> use 'vdsmjsonrpcclient' module to communicate with vdsm, but it cannot >>> find this module, probably because it is an older version. >>> >>> Updating MOM to version 0.5.12 should fix it. >>> >>> >>> Regards, >>> Andrej >>> >>> On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>> wrote: >>> >>>> FWIW: >>>> >>>> This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) >>>> Xeon(R) CPU E3-1220 V2 @ 3.10GHz >>>> >>>> Network is bonded fail-over. >>>> >>>> Regards, >>>> Brett >>>> >>>> On 18 July 2018 at 13:15, Francesco Romani <fromani(a)redhat.com&gt; wrote: >>>> >>>>> Thanks! >>>>> >>>>> >>>>> On 07/18/2018 02:11 PM, Maton, Brett wrote: >>>>> >>>>>> Sure no problem, mom log attached. >>>>>> >>>>> >>>>> It seems to me that MOM is restarting *very* often: >>>>> >>>>> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >>>>> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >>>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>>> >>>>> And I totally not expect this. Martin, Andrej, is this the expected >>>>> behaviour? >>>>> >>>>> >>>>> Bests, >>>>> >>>>> >>>>> >>>>> -- >>>>> Francesco Romani >>>>> Senior SW Eng., Virtualization R&D >>>>> Red Hat >>>>> IRC: fromani github: @fromanirh >>>>> >>>>> >>>> >>

Thanks, Cluster is all installed from pre-release, maybe I managed to get an iffy rpm.... On 18 July 2018 at 15:50, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: > Yes, copying it from another host with mom version 0.5.12 is enough. > > I'm not sure why it was not installed. The file is in the rpm. > > On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb(a)ltresources.co.uk&gt; > wrote: > >> Bingo >> >> How could that file not be installed/deployed ? >> >> >> Should I simply copy it from one of the other hosts to make the message >> go away ? >> >> On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: >> >>> Maybe the python module is missing. Can you check if this file exists? >>> >>> /usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/vd >>> smjsonrpcclientInterface.py >>> >>> >>> Andrej >>> >>> On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>> wrote: >>> >>>> I just checked the mom version, it's already at 0.5.12 >>>> >>>> # rpm -qa mom >>>> mom-0.5.12-1.el7.centos.noarch >>>> >>>> All the physical hosts are fully updated /patched. >>>> >>>> Regards, >>>> Brett >>>> >>>> On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: >>>> >>>>> Hi, >>>>> >>>>> Looking at MOM logs, the problem seems to be that MOM is configured >>>>> to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it >>>>> cannot find this module, probably because it is an older version. >>>>> >>>>> Updating MOM to version 0.5.12 should fix it. >>>>> >>>>> >>>>> Regards, >>>>> Andrej >>>>> >>>>> On Wed, 18 Jul 2018 at 14:41, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>>>> wrote: >>>>> >>>>>> FWIW: >>>>>> >>>>>> This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) >>>>>> Xeon(R) CPU E3-1220 V2 @ 3.10GHz >>>>>> >>>>>> Network is bonded fail-over. >>>>>> >>>>>> Regards, >>>>>> Brett >>>>>> >>>>>> On 18 July 2018 at 13:15, Francesco Romani <fromani(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>> On 07/18/2018 02:11 PM, Maton, Brett wrote: >>>>>>> >>>>>>>> Sure no problem, mom log attached. >>>>>>>> >>>>>>> >>>>>>> It seems to me that MOM is restarting *very* often: >>>>>>> >>>>>>> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >>>>>>> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >>>>>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>>>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>>>>> >>>>>>> And I totally not expect this. Martin, Andrej, is this the expected >>>>>>> behaviour? >>>>>>> >>>>>>> >>>>>>> Bests, >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Francesco Romani >>>>>>> Senior SW Eng., Virtualization R&D >>>>>>> Red Hat >>>>>>> IRC: fromani github: @fromanirh >>>>>>> >>>>>>> >>>>>> >>>> >>

I think the upgrade may not have affected the mom package, because no new mom rpm version was released for a few months. I'm not sure why the import fails even if the file is there. As a workaround, mom can be configured to use the older (deprecated) json rpc interface to communicate with vdsm. There is no difference in functionality. This can be done by editing /etc/vdsm/mom.conf. Set the 'hypervisor-interface' config value to 'vdsmjsonrpcbulk' insted of 'vdsmjsonrpcclient'. On Wed, 25 Jul 2018 at 08:56, Maton, Brett <matonb(a)ltresources.co.uk&gt; wrote: > I upgraded my test cluster to 4.2.5.2-1 last night (hosts rebooted after > update) and I've started to see the error again. > > As before it seems to be limited to the one host, I've checked that > > /usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/ > vdsmjsonrpcclientInterface.py > > exists, which it does but there are no matching .pyc / .pyo files which > suggests that it's not been used? > > File permissions and SELinux contexts are the same on all files in the > HypervisorInterfaces directory. > > 0644 root root system_u:object_r:lib_t:s0 > > vdsm / mom log from time period attached. > > > On 18 July 2018 at 16:10, Maton, Brett <matonb(a)ltresources.co.uk&gt; wrote: > >> Thanks, >> >> Cluster is all installed from pre-release, maybe I managed to get an >> iffy rpm.... >> >> On 18 July 2018 at 15:50, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: >> >>> Yes, copying it from another host with mom version 0.5.12 is enough. >>> >>> I'm not sure why it was not installed. The file is in the rpm. >>> >>> On Wed, 18 Jul 2018 at 16:38, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>> wrote: >>> >>>> Bingo >>>> >>>> How could that file not be installed/deployed ? >>>> >>>> >>>> Should I simply copy it from one of the other hosts to make the >>>> message go away ? >>>> >>>> On 18 July 2018 at 15:30, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: >>>> >>>>> Maybe the python module is missing. Can you check if this file exists? >>>>> >>>>> /usr/lib/python2.7/site-packages/mom/HypervisorInterfaces/ >>>>> vdsmjsonrpcclientInterface.py >>>>> >>>>> >>>>> Andrej >>>>> >>>>> On Wed, 18 Jul 2018 at 16:04, Maton, Brett <matonb(a)ltresources.co.uk&gt; >>>>> wrote: >>>>> >>>>>> I just checked the mom version, it's already at 0.5.12 >>>>>> >>>>>> # rpm -qa mom >>>>>> mom-0.5.12-1.el7.centos.noarch >>>>>> >>>>>> All the physical hosts are fully updated /patched. >>>>>> >>>>>> Regards, >>>>>> Brett >>>>>> >>>>>> On 18 July 2018 at 14:08, Andrej Krejcir <akrejcir(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Looking at MOM logs, the problem seems to be that MOM is configured >>>>>>> to use 'vdsmjsonrpcclient' module to communicate with vdsm, but it >>>>>>> cannot find this module, probably because it is an older version. >>>>>>> >>>>>>> Updating MOM to version 0.5.12 should fix it. >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Andrej >>>>>>> >>>>>>> On Wed, 18 Jul 2018 at 14:41, Maton, Brett < >>>>>>> matonb(a)ltresources.co.uk&gt; wrote: >>>>>>> >>>>>>>> FWIW: >>>>>>>> >>>>>>>> This test cluster is 3x HP MicroServer Gen 8 16GB RAM, Intel(R) >>>>>>>> Xeon(R) CPU E3-1220 V2 @ 3.10GHz >>>>>>>> >>>>>>>> Network is bonded fail-over. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Brett >>>>>>>> >>>>>>>> On 18 July 2018 at 13:15, Francesco Romani <fromani(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> >>>>>>>>> On 07/18/2018 02:11 PM, Maton, Brett wrote: >>>>>>>>> >>>>>>>>>> Sure no problem, mom log attached. >>>>>>>>>> >>>>>>>>> >>>>>>>>> It seems to me that MOM is restarting *very* often: >>>>>>>>> >>>>>>>>> 2018-07-18 06:07:04,909 - mom - INFO - MOM starting >>>>>>>>> 2018-07-18 06:07:20,407 - mom - INFO - MOM starting >>>>>>>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>>>>>>> 2018-07-18 06:07:30,658 - mom - INFO - MOM starting >>>>>>>>> >>>>>>>>> And I totally not expect this. Martin, Andrej, is this the >>>>>>>>> expected behaviour? >>>>>>>>> >>>>>>>>> >>>>>>>>> Bests, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Francesco Romani >>>>>>>>> Senior SW Eng., Virtualization R&D >>>>>>>>> Red Hat >>>>>>>>> IRC: fromani github: @fromanirh >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >> >