cpuTune stats overflowing xmlrpc
Hi Kobi, I have found a problem with your vdsm implementation of the cpuTune stats. When using vdsClient I see the following error: sudo vdsClient -s 0 getVmStats 359a9045-34e2-40d2-ba5f-ba614421a624 Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2668, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 459, in do_getVmStats return self.ExecAndExit(self.s.getVmStats(vmId)) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/site-packages/vdsm/vdscli.py", line 41, in wrapped_parse_response return old_parse_response(*args, **kwargs) File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.OverflowError'>:long int exceeds XML-RPC limits"> You need to convert Long ints to strings to avoid overflows. -- Adam Litke
Hi Adam, can you identify the actual value that causes this? The cpu code added only percentages and vcpuQuota/Period.. all those numbers are pretty low. I do not see anything that would be more than 2^32 big (4 bilion iirc) in that code.. -- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message -----
Hi Kobi,
I have found a problem with your vdsm implementation of the cpuTune stats. When using vdsClient I see the following error:
sudo vdsClient -s 0 getVmStats 359a9045-34e2-40d2-ba5f-ba614421a624 Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2668, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 459, in do_getVmStats return self.ExecAndExit(self.s.getVmStats(vmId)) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/site-packages/vdsm/vdscli.py", line 41, in wrapped_parse_response return old_parse_response(*args, **kwargs) File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close raise Fault(**self._stack[0]) Fault: <Fault 1: "<type 'exceptions.OverflowError'>:long int exceeds XML-RPC limits">
You need to convert Long ints to strings to avoid overflows.
-- Adam Litke
On 19/06/14 10:35 -0400, Martin Sivak wrote:
Hi Adam,
can you identify the actual value that causes this?
The cpu code added only percentages and vcpuQuota/Period.. all those numbers are pretty low.
I do not see anything that would be more than 2^32 big (4 bilion iirc) in that code..
Thread-19::DEBUG::2014-06-19 11:19:09,655::BindingXMLRPC::1114::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'displayInfo': [{'tlsPort': u'5900', 'ipAddress': '192.168.2.101', 'type': 'spice', 'port': '-1'}], 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '1371', 'session': 'Unknown', 'vmType': 'kvm', 'timeOffset': '0', 'balloonInfo': {'balloon_max': '1048576', 'balloon_target': 0, 'balloon_cur': '1048576', 'balloon_min': '1048576'}, 'pauseCode': 'NOERR', 'network': {u'vnet0': {'macAddr': '00:1a:4a:3f:44:ad', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmJobs': {}, 'cpuUser': '1.68', 'elapsedTime': '21', 'displayType': 'qxl', 'cpuSys': '0.33', 'appsList': [], 'vcpuCount': '1', 'clientIp': '', 'hash': '-8539662102253950265', 'vmId': '359a9045-34e2-40d2-ba5f-ba614421a624', 'displayIp': '192.168.2.101', 'vcpuPeriod': 100000L, 'displayPort': '-1', 'vcpuQuota': 18446744073709551615L, 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '197120', 'writeLatency': '0', 'imageID': 'cf6594ae-ac06-47ea-931e-9f280c271a05', 'flushLatency': '0', 'truesize': '200704'}, u'hdc': {'flushLatency': '0', 'readLatency': '0', 'truesize': '0', 'apparentsize': '0', 'writeLatency': '0'}}, 'monitorResponse': '0', 'statsAge': '0.85', 'username': 'Unknown', 'status': 'Powering up', 'guestCPUCount': -1, 'guestIPs': '', 'displaySecurePort': u'5900'}]} 'vcpuQuota': 18446744073709551615L <-- smoking gun -- Adam Litke
Yep you are right, The value can really get high: quota The optional quota element specifies the maximum allowed bandwidth(unit: microseconds). A domain with quota as any negative value indicates that the domain has infinite bandwidth, which means that it is not bandwidth controlled. The value should be in range [1000, 18446744073709551] or less than 0. A quota with value 0 means no value. You can use this feature to ensure that all vcpus run at the same speed. Only QEMU driver support since 0.9.4, LXC since 0.9.10 -- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message -----
On 19/06/14 10:35 -0400, Martin Sivak wrote:
Hi Adam,
can you identify the actual value that causes this?
The cpu code added only percentages and vcpuQuota/Period.. all those numbers are pretty low.
I do not see anything that would be more than 2^32 big (4 bilion iirc) in that code..
Thread-19::DEBUG::2014-06-19 11:19:09,655::BindingXMLRPC::1114::vds::(wrapper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'displayInfo': [{'tlsPort': u'5900', 'ipAddress': '192.168.2.101', 'type': 'spice', 'port': '-1'}], 'memUsage': '0', 'acpiEnable': 'true', 'guestFQDN': '', 'pid': '1371', 'session': 'Unknown', 'vmType': 'kvm', 'timeOffset': '0', 'balloonInfo': {'balloon_max': '1048576', 'balloon_target': 0, 'balloon_cur': '1048576', 'balloon_min': '1048576'}, 'pauseCode': 'NOERR', 'network': {u'vnet0': {'macAddr': '00:1a:4a:3f:44:ad', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmJobs': {}, 'cpuUser': '1.68', 'elapsedTime': '21', 'displayType': 'qxl', 'cpuSys': '0.33', 'appsList': [], 'vcpuCount': '1', 'clientIp': '', 'hash': '-8539662102253950265', 'vmId': '359a9045-34e2-40d2-ba5f-ba614421a624', 'displayIp': '192.168.2.101', 'vcpuPeriod': 100000L, 'displayPort': '-1', 'vcpuQuota': 18446744073709551615L, 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '197120', 'writeLatency': '0', 'imageID': 'cf6594ae-ac06-47ea-931e-9f280c271a05', 'flushLatency': '0', 'truesize': '200704'}, u'hdc': {'flushLatency': '0', 'readLatency': '0', 'truesize': '0', 'apparentsize': '0', 'writeLatency': '0'}}, 'monitorResponse': '0', 'statsAge': '0.85', 'username': 'Unknown', 'status': 'Powering up', 'guestCPUCount': -1, 'guestIPs': '', 'displaySecurePort': u'5900'}]}
'vcpuQuota': 18446744073709551615L <-- smoking gun
-- Adam Litke
participants (2)
-
Adam Litke -
Martin Sivak