On 07/18/2018 07:24 AM, Maton, Brett wrote:

Thanks Francesco,

  Log attached.

Interestings, it seems the fault comes from MOM:

2018-07-18 06:07:22,836+0100 INFO  (jsonrpc/3) [api.host] FINISH getStats error=<Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'statistics'"> from=::ffff:10.11.30.253,58840 (api:50)
2018-07-18 06:07:22,836+0100 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:611)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod
    result = fn(*methodArgs)
  File "<string>", line 2, in getStats
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1402, in getStats
    sampling.host_samples.stats())}
  File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 75, in get_stats
    ret.update(cif.mom.getKsmStats())
  File "/usr/lib/python2.7/site-packages/vdsm/momIF.py", line 75, in getKsmStats
    stats = self._mom.getStatistics()['host']
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1306, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1482, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 794, in close
    raise Fault(**self._stack[0])
Fault: <Fault 1: "<type 'exceptions.AttributeError'>:'NoneType' object has no attribute 'statistics'">
2018-07-18 06:07:22,836+0100 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getStats failed (error -32603) in 0.01 seconds (__init__:573)
2018-07-18 06:07:22,909+0100 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)
2018-07-18 06:07:22,912+0100 INFO  (jsonrpc/4) [api.virt] START getStats() from=::1,60058, vmId=a1af259b-7917-49ac-8bcc-eb4cc302e723 (api:46)
2018-07-18 06:07:22,912+0100 INFO  (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': u'a1af259b-7917-49ac-8bcc-eb4cc302e723'} (api:127)
2018-07-18 06:07:22,912+0100 INFO  (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'a1af259b-7917-49ac-8bcc-eb4cc302e723'}", 'code': 1}} from=::1,60058, vmId=a1af259b-7917-49ac-8bcc-eb4cc302e723 (api:52)
2018-07-18 06:07:22,912+0100 INFO  (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:573)
2018-07-18 06:07:23,043+0100 INFO  (jsonrpc/6) [root] /usr/libexec/vdsm/hooks/after_get_caps/50_openstacknet: rc=0 err= (hooks:110)

It could be a (benign) race on shutdown, but better continue the investigation. Could you please also chare the mom.log corresponding to the timeframe you already provided?

Thanks,

--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh