[Engine-devel] Consider iowait add into usage CPU percentage

11 Mar 2014

      --_000_B63C858E777679458338A30A991BB524015C66EBG1W3780americas_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi All,

On engine core, The usage CPU percentage is calculated by %sys + %usr

class VdsBrokerObjectsBuilder
function updateVDSStatisticsData
vds.setCpuSys(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_sys));
        vds.setCpuUser(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_us=
er));
        if (vds.getCpuSys() !=3D null && vds.getCpuUser() !=3D null) {
            vds.setUsageCpuPercent((int) (vds.getCpuSys() + vds.getCpuUser(=
)));
        }

On vdsm, The %sys, %usr and %idle is calculated like below workflow,

class API
function getStats
decStats =3D self._cif._hostStats.get()
class clientIF
function __init__
                   self._hostStats =3D sampling.HostStatsThread(log=3Dlog)
        self._hostStats.start()
class HostStatsThread
function get
                   hs0, hs1 =3D self._samples[0], self._samples[-1]
                   ...
                   jiffies =3D (hs1.totcpu.user - hs0.totcpu.user) % (2 ** =
32)
        stats['cpuUser'] =3D jiffies / interval / self._ncpus
        jiffies =3D (hs1.totcpu.sys - hs0.totcpu.sys) % (2 ** 32)
        stats['cpuSys'] =3D jiffies / interval / self._ncpus
        stats['cpuIdle'] =3D max(0.0, 100.0 - stats['cpuUser'] - stats['cpu=
Sys'])
class HostSample
function __init__
                   self.totcpu =3D TotalCpuSample()
class TotalCpuSample
function __init__
         self.user, userNice, self.sys, self.idle =3D \
            map(int, file('/proc/stat').readline().split()[1:5])
        self.user +=3D userNice

Question1: Why stats['cpuIdle'] do not use the sampling data totcpu.idle fo=
r calculating?

Question2: There is another data named iowait in /proc/stat, do we need to =
consider add this into usage CPU percentage for calculating?

Best Regards,
Jason Liao

--_000_B63C858E777679458338A30A991BB524015C66EBG1W3780americas_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
...
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
	{font-family:SimSun;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"HP Simplified";
	panose-1:2 11 6 4 2 2 4 2 2 4;}
@font-face
	{font-family:"YaHei Consolas Hybrid";
	panose-1:2 11 5 9 2 2 4 2 2 4;}
@font-face
	{font-family:"\@YaHei Consolas Hybrid";
	panose-1:2 11 5 9 2 2 4 2 2 4;}
@font-face
	{font-family:SimSun;
	panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	text-align:justify;
	text-justify:inter-ideograph;
	font-size:10.5pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"HP Simplified","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
/* Page Definitions */
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
	{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"ZH-CN" link=3D"#0563C1" vlink=3D"#954F72" style=3D"text-justi=
fy-trim:punctuation">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">Hi All,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">On engine core, The usage CPU perce=
ntage is calculated by %sys + %usr<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">class VdsBrokerObjectsBuilder
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">function updateVDSStatisticsData<o:=
p></o:p></span></p>
<p class=3D"MsoNormal" style=3D"margin-left:21.0pt;text-indent:21.0pt"><spa=
n lang=3D"EN-US" style=3D"font-family:"HP Simplified","sans-=
serif"">vds.setCpuSys(AssignDoubleValue(xmlRpcStruct, VdsProperties.cp=
u_sys));<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  vds.setCpuUser(AssignDoubleValue(xmlRpcStruct, VdsProperties.cpu_us=
er));<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  if (vds.getCpuSys() !=3D null && vds.getCpuUser() !=3D null=
) {<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;      vds.setUsageCpuPercent((int) (vds.getCpuSys=
() + vds.getCpuUser()));<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  }<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">On vdsm, The %sys, %usr and %idle i=
s calculated like below workflow,<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">class API<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">function getStats<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal" style=3D"margin-left:21.0pt;text-indent:21.0pt"><spa=
n lang=3D"EN-US" style=3D"font-family:"HP Simplified","sans-=
serif"">decStats =3D self._cif._hostStats.get()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">class clientIF<o:p></o:p></span></p=
...
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">function __init__<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;             s=
elf._hostStats =3D sampling.HostStatsThread(log=3Dlog)<o:p></o:p></span></p=
...
...
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  stats['cpuUser'] =3D jiffies / interval / self._ncpus<o:p></o:p></s=
...
</span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  stats['cpuSys'] =3D jiffies / interval / self._ncpus<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  stats['cpuIdle'] =3D max(0.0, 100.0 - stats['cpuUser'] - stats['cpu=
Sys'])<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">class HostSample<o:p></o:p></span><=
/p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">function __init__<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;             s=
elf.totcpu =3D TotalCpuSample()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">class TotalCpuSample<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">function __init__<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;   self.user, userNice, self.sys, self.idle =3D \<o:p></o:p></sp=
an></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;      map(int, file('/proc/stat').readline().spli=
t()[1:5])<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  self.user +=3D userNice<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">Question1: Why stats[‘cpuIdle=
’] do not use the sampling data totcpu.idle for calculating?<o:p></o:=
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  self._hostStats.start()<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">class HostStatsThread<o:p></o:p></s=
pan></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">function get<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;             h=
s0, hs1 =3D self._samples[0], self._samples[-1]<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;             &=
#8230;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;             j=
iffies =3D (hs1.totcpu.user - hs0.totcpu.user) % (2 ** 32)<o:p></o:p></span=
pan></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">      =
;  jiffies =3D (hs1.totcpu.sys - hs0.totcpu.sys) % (2 ** 32)<o:p></o:p=
p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif"">Question2: There is another data na=
med iowait in /proc/stat, do we need to consider add this into usage CPU pe=
rcentage for calculating?<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S=
implified","sans-serif""><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"font-size:9.0pt;fon=
t-family:"HP Simplified","sans-serif";color:black">Best=
 Regards,<br>
</span></b><span lang=3D"EN-US" style=3D"font-size:9.0pt;font-family:"=
HP Simplified","sans-serif";color:#717172">Jason Liao</span>=
<span lang=3D"EN-US"><o:p></o:p></span></p>
<p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>

--_000_B63C858E777679458338A30A991BB524015C66EBG1W3780americas_--

[Engine-devel] Consider iowait add into usage CPU percentage

Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC)