[Engine-devel] FW: Discussion about NUMA feature and CPU pinning

=C4=EA</span><span lang=3D"EN-US" style=3D"font-size:11.0pt">4</span><span=
</span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif""><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif"">There are some operations in list:<= o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:18.0pt;text-indent:-18.0=
</span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:-21.0=
--_000_B63C858E777679458338A30A991BB524015F8CAEG2W2441americas_ Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 Rm9yd2FyZCB0byBwdWJsaWMgZ3JvdXANCg0KQmVzdCBSZWdhcmRzLA0KSmFzb24gTGlhbw0KDQpG cm9tOiBMaWFvLCBDaHVhbiAoSmFzb24gTGlhbywgSFBzZXJ2ZXJzLUNvcmUtT0UtUFNDKQ0KU2Vu dDogMjAxNMTqNNTCMcjVIDE2OjU2DQpUbzogJ0dpbGFkIENoYXBsaWsnDQpDYzogRG9yb24gRmVk aXVjazsgVmlub2QsIENoZWd1OyBMaWFuZywgU2hhbmctQ2h1biAoRGF2aWQgTGlhbmcsIEhQc2Vy dmVycy1Db3JlLU9FLVBTQyk7IFNoaSwgWGlhby1MZWkgKEJydWNlLCBIUCBTZXJ2ZXJzLVBTQy1D USk7ICdkZXZlbEBvdmlydC5vcmcnDQpTdWJqZWN0OiBEaXNjdXNzaW9uIGFib3V0IE5VTUEgZmVh dHVyZSBhbmQgQ1BVIHBpbm5pbmcNCg0KSGkgR2lsYWQsDQoNCldoZW4gSSBkZWZpbmUgVmRjQWN0 aW9uVHlwZSB0byBtYW5hZ2UgVmlydHVhbCBOVU1BIG5vZGUgYW5kIHBpbiBWaXJ0dWFsIE5VTUEg bm9kZSB0byBob3N0IE5VTUEgbm9kZS4NCg0KVGhlcmUgYXJlIHNvbWUgb3BlcmF0aW9ucyBpbiBs aXN0Og0KDQoxLiAgICAgIE5ldyBWaXJ0dWFsIE5VTUEgbm9kZSAoIHNldCB2Y3B1cyBjb3VudCwg c2V0IHRvdGFsIG1lbW9yeSApDQoNCjIuICAgICAgUGluIFZpcnR1YWwgTlVNQSBub2RlIHRvIGhv c3QgTlVNQSBub2RlICggc2F2ZSBob3N0IE5VTUEgbm9kZSBJRCBpbnRvIFZpcnR1YWwgTlVNQSBu b2RlICkNCg0KMy4gICAgICBTYXZlIGEgVk2hr3MgYWxsIFZpcnR1YWwgTlVNQSBub2RlDQoNCmEp ICAgICAgICAgQ2FsY3VsYXRlIFZNIHBpbiB0byBob3N0IHByb3BlcnR5IGZyb20gaG9zdCBOVU1B IG5vZGWhr3MgaG9zdCBJRC4NCg0KYikgICAgICAgICBDYWxjdWxhdGUgVk0gTlVNQSB0dW5pbmcg bm9kZXNldCBmcm9tIGFsbCBWaXJ0dWFsIE5VTUEgbm9kZaGvcyBwaW4gdG8gaG9zdCBOVU1BIG5v ZGUgSUQuDQoNCmMpICAgICAgICAgQ2FsY3VsYXRlIFZNIENQVSBwaW5uaW5nIGZyb20gYWxsIFZp cnR1YWwgTlVNQSBub2Rloa9zIHZjcHVzIGFuZCByZWxhdGVkIHBpbiB0byBob3N0IE5VTUEgbm9k ZSBjcHVzDQoNCmUuZy4NCg0KaG9zdCBOVU1BIG5vZGUgMCBjcHVzIDAsIDEsIDIsIDMNCg0KaG9z dCBOVU1BIG5vZGUgMSBjcHVzIDQsIDUsIDYsIDcNCg0KaG9zdCBOVU1BIG5vZGUgMiBjcHVzIDgs IDksIDEwLCAxMQ0KDQpob3N0IE5VTUEgbm9kZSAzIGNwdXMgMTIsIDEzLCAxNCwgMTUNCg0Kdmly dHVhbCBOVU1BIG5vZGUgMCB2Y3B1cyBjb3VudCAyDQoNCnZpcnR1YWwgTlVNQSBub2RlIDEgdmNw dXMgY291bnQgMg0KDQoNCg0KcGluIHZpcnR1YWwgTlVNQSBub2RlIDAgdG8gaG9zdCBOVU1BIG5v ZGUgMQ0KDQpwaW4gdmlydHVhbCBOVU1BIG5vZGUgMSB0byBob3N0IE5VTUEgbm9kZSAzDQoNCg0K DQpiLiBDYWxjdWxhdGUgcmVzdWx0OiAxLDMNCg0KYy4gQ2FsY3VsYXRlIHJlc3VsdDogMCM0LDUs Niw3XzEjNCw1LDYsN18yIzEyLDEzLDE0LDE1XzMjMTIsMTMsMTQsMTUNCg0KRGlzY3Vzc2lvbjoN Cg0KMS4gICAgICBEbyB3ZSBuZWVkIHRoZXNlIG9wZXJhdGlvbnMsIGVzcGVjaWFsbHkgbWV0aG9k IGMuID8NCg0KMi4gICAgICBJZiB0aGUgYW5zd2VyIGlzIFllcywgdGhpcyBvcGVyYXRpb24gd2ls bCByZXBsYWNlIHRoZSBDcHUgcGlubmluZyBjb25maWd1cmF0aW9uIGZyb20gVk0gQ3B1IHBpbmlu ZyBpbnB1dCwgRG8gd2UgbmVlZCB0byBub3RpZnkgdGhlIG93bmVyID8NCg0KQmVzdCBSZWdhcmRz LA0KSmFzb24gTGlhbw0KDQo= --_000_B63C858E777679458338A30A991BB524015F8CAEG2W2441americas_ Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dgb2312"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:SimSun; panose-1:2 1 6 0 3 1 1 1 1 1;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:SimSun; panose-1:2 1 6 0 3 1 1 1 1 1;} @font-face {font-family:"YaHei Consolas Hybrid"; panose-1:2 11 5 9 2 2 4 2 2 4;} @font-face {font-family:"\@YaHei Consolas Hybrid"; panose-1:2 11 5 9 2 2 4 2 2 4;} @font-face {font-family:"HP Simplified"; panose-1:2 11 6 4 2 2 4 2 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; font-size:10.5pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {mso-style-priority:34; margin:0cm; margin-bottom:.0001pt; text-align:justify; text-justify:inter-ideograph; text-indent:21.0pt; font-size:10.5pt; font-family:"Calibri","sans-serif";} span.EmailStyle18 {mso-style-type:personal; font-family:"HP Simplified","sans-serif"; color:windowtext;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"HP Simplified","sans-serif"; color:#7030A0;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 90.0pt 72.0pt 90.0pt;} div.WordSection1 {page:WordSection1;} /* List Definitions */ @list l0 {mso-list-id:1736854702; mso-list-type:hybrid; mso-list-template-ids:1732427620 -992544284 67698713 67698715 67698703 676= 98713 67698715 67698703 67698713 67698715;} @list l0:level1 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:18.0pt; text-indent:-18.0pt;} @list l0:level2 {mso-level-number-format:alpha-lower; mso-level-text:"%2\)"; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:42.0pt; text-indent:-21.0pt;} @list l0:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:63.0pt; text-indent:-21.0pt;} @list l0:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:84.0pt; text-indent:-21.0pt;} @list l0:level5 {mso-level-number-format:alpha-lower; mso-level-text:"%5\)"; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:105.0pt; text-indent:-21.0pt;} @list l0:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:126.0pt; text-indent:-21.0pt;} @list l0:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:147.0pt; text-indent:-21.0pt;} @list l0:level8 {mso-level-number-format:alpha-lower; mso-level-text:"%8\)"; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:168.0pt; text-indent:-21.0pt;} @list l0:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:189.0pt; text-indent:-21.0pt;} @list l1 {mso-list-id:2035881049; mso-list-type:hybrid; mso-list-template-ids:440727406 1092133814 67698713 67698715 67698703 6769= 8713 67698715 67698703 67698713 67698715;} @list l1:level1 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:18.0pt; text-indent:-18.0pt;} @list l1:level2 {mso-level-number-format:alpha-lower; mso-level-text:"%2\)"; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:42.0pt; text-indent:-21.0pt;} @list l1:level3 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:63.0pt; text-indent:-21.0pt;} @list l1:level4 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:84.0pt; text-indent:-21.0pt;} @list l1:level5 {mso-level-number-format:alpha-lower; mso-level-text:"%5\)"; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:105.0pt; text-indent:-21.0pt;} @list l1:level6 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:126.0pt; text-indent:-21.0pt;} @list l1:level7 {mso-level-tab-stop:none; mso-level-number-position:left; margin-left:147.0pt; text-indent:-21.0pt;} @list l1:level8 {mso-level-number-format:alpha-lower; mso-level-text:"%8\)"; mso-level-tab-stop:none; mso-level-number-position:left; margin-left:168.0pt; text-indent:-21.0pt;} @list l1:level9 {mso-level-number-format:roman-lower; mso-level-tab-stop:none; mso-level-number-position:right; margin-left:189.0pt; text-indent:-21.0pt;} ol {margin-bottom:0cm;} ul {margin-bottom:0cm;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3D"ZH-CN" link=3D"#0563C1" vlink=3D"#954F72" style=3D"text-justi= fy-trim:punctuation"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif";color:#7030A0">Forward to public gro= up<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif";color:#7030A0"><o:p> </o:p></sp= an></p> <div> <p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"font-size:9.0pt;fon= t-family:"HP Simplified","sans-serif";color:black">Best= Regards,<br> </span></b><span lang=3D"EN-US" style=3D"font-size:9.0pt;font-family:"= HP Simplified","sans-serif";color:#717172">Jason Liao</span>= <span lang=3D"EN-US" style=3D"color:#7030A0"><o:p></o:p></span></p> </div> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif";color:#7030A0"><o:p> </o:p></sp= an></p> <div> <div style=3D"border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm = 0cm 0cm"> <p class=3D"MsoNormal" align=3D"left" style=3D"text-align:left"><b><span la= ng=3D"EN-US" style=3D"font-size:11.0pt">From:</span></b><span lang=3D"EN-US= " style=3D"font-size:11.0pt"> Liao, Chuan (Jason Liao, HPservers-Core-OE-PS= C) <br> <b>Sent:</b> 2014</span><span style=3D"font-size:11.0pt;font-family:SimSun"= style=3D"font-size:11.0pt;font-family:SimSun">=D4=C2</span><span lang=3D"E= N-US" style=3D"font-size:11.0pt">1</span><span style=3D"font-size:11.0pt;fo= nt-family:SimSun">=C8=D5</span><span lang=3D"EN-US" style=3D"font-size:11.0= pt"> 16:56<br> <b>To:</b> 'Gilad Chaplik'<br> <b>Cc:</b> Doron Fediuck; Vinod, Chegu; Liang, Shang-Chun (David Liang, HPs= ervers-Core-OE-PSC); Shi, Xiao-Lei (Bruce, HP Servers-PSC-CQ); 'devel@ovirt= .org'<br> <b>Subject:</b> Discussion about NUMA feature and CPU pinning<o:p></o:p></s= pan></p> </div> </div> <p class=3D"MsoNormal" align=3D"left" style=3D"text-align:left"><span lang= =3D"EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif"">Hi Gilad,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif""><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif"">When I define VdcActionType to mana= ge Virtual NUMA node and pin Virtual NUMA node to host NUMA node.<o:p></o:p= pt;mso-list:l0 level1 lfo2"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">1.<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">New Virtual NUMA node ( set= vcpus count, set total memory )<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:18.0pt;text-indent:-18.0= pt;mso-list:l0 level1 lfo2"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">2.<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">Pin Virtual NUMA node to ho= st NUMA node ( save host NUMA node ID into Virtual NUMA node )<o:p></o:p></= span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:18.0pt;text-indent:-18.0= pt;mso-list:l0 level1 lfo2"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">3.<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">Save a VM=A1=AFs all Virtua= l NUMA node<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:-21.0= pt;mso-list:l0 level2 lfo2"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">a)<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">Calculate VM pin to host pr= operty from host NUMA node=A1=AFs host ID.<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:-21.0= pt;mso-list:l0 level2 lfo2"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">b)<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">Calculate VM NUMA tuning no= deset from all Virtual NUMA node=A1=AFs pin to host NUMA node ID.<o:p></o:p= pt;mso-list:l0 level2 lfo2"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">c)<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">Calculate VM CPU pinning fr= om all Virtual NUMA node=A1=AFs vcpus and related pin to host NUMA node cpu= s<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">e.g. <o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">host NUMA node 0 cpus 0, 1, 2, 3<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">host NUMA node 1 cpus 4, 5, 6, 7<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">host NUMA node 2 cpus 8, 9, 10, 11<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">host NUMA node 3 cpus 12, 13, 14, 15<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">virtual NUMA node 0 vcpus count 2<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">virtual NUMA node 1 vcpus count 2<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif""><o:p> </o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">pin virtual NUMA node 0 to host NUMA node 1<o:p></o:p></sp= an></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">pin virtual NUMA node 1 to host NUMA node 3<o:p></o:p></sp= an></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif""><o:p> </o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">b. Calculate result: 1,3<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:42.0pt;text-indent:0cm">= <span lang=3D"EN-US" style=3D"font-family:"HP Simplified","s= ans-serif"">c. Calculate result: 0#4,5,6,7_1#4,5,6,7_2#12,13,14,15_3#1= 2,13,14,15<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif""><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif"">Discussion:<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:18.0pt;text-indent:-18.0= pt;mso-list:l1 level1 lfo4"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">1.<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">Do we need these operations= , especially method c. ?<o:p></o:p></span></p> <p class=3D"MsoListParagraph" style=3D"margin-left:18.0pt;text-indent:-18.0= pt;mso-list:l1 level1 lfo4"> <![if !supportLists]><span lang=3D"EN-US" style=3D"font-family:"HP Sim= plified","sans-serif""><span style=3D"mso-list:Ignore">2.<sp= an style=3D"font:7.0pt "Times New Roman"">  = ; </span></span></span><![endif]><span lang=3D"EN-US" style=3D"font-family:&q= uot;HP Simplified","sans-serif"">If the answer is Yes, this = operation will replace the Cpu pinning configuration from VM Cpu pining inp= ut, Do we need to notify the owner ?<o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US" style=3D"font-family:"HP S= implified","sans-serif""><o:p> </o:p></span></p> <p class=3D"MsoNormal"><b><span lang=3D"EN-US" style=3D"font-size:9.0pt;fon= t-family:"HP Simplified","sans-serif";color:black">Best= Regards,<br> </span></b><span lang=3D"EN-US" style=3D"font-size:9.0pt;font-family:"= HP Simplified","sans-serif";color:#717172">Jason Liao</span>= <span lang=3D"EN-US"><o:p></o:p></span></p> <p class=3D"MsoNormal"><span lang=3D"EN-US"><o:p> </o:p></span></p> </div> </body> </html> --_000_B63C858E777679458338A30A991BB524015F8CAEG2W2441americas_--
participants (1)
-
Liao, Chuan (Jason Liao, HPservers-Core-OE-PSC)