- Users - oVirt List Archives

Data Operations On Any Host
by Cryptic 09 Apr '18

09 Apr '18

--_000_SL2P216MB0906F39F025F11FBE21AFB72ACA00SL2P216MB0906KORP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, In relation to the change made to distribute data operations between all th= e hosts in a data center rather than burden the SPM. I am having troubles f= inding information on this and need assistance to prevent this happening on= my development oVirt 4.2 system. The issue I have is that I have a cluster= which hosts all the storage volumes using gluster and they have 10G NICs. = I also have a separate cluster which is virtualisation only and each host o= nly has 3 x 1G aggregated NICs. When I perform disk moves between storage d= omains it often uses one of the virtualisation hosts which drastically incr= eases the time taken to move the disk. Can I restrict these types of operat= ions to a set of hosts or turn it off altogether so that it just uses the S= PM like it used to in the past. Distributing it is a great feature but unfo= rtunately is no good in my current setup. Regards, Jeremy --_000_SL2P216MB0906F39F025F11FBE21AFB72ACA00SL2P216MB0906KORP_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"></style> </head> <body dir=3D"ltr"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Helvetica,sans-serif;" dir=3D"ltr"> <p></p> <div>Hi,</div> <div><br> </div> <div>In relation to the change made to distribute data operations between a= ll the hosts in a data center rather than burden the SPM. I am having troub= les finding information on this and need assistance to prevent this happeni= ng on my development oVirt 4.2 system. The issue I have is that I have a cluster which hosts all the storage volu= mes using gluster and they have 10G NICs. I also have a separate cluster wh= ich is virtualisation only and each host only has 3 x 1G aggregated NICs. W= hen I perform disk moves between storage domains it often uses one of the virtualisation hosts which drasti= cally increases the time taken to move the disk. Can I restrict these types= of operations to a set of hosts or turn it off altogether so that it just = uses the SPM like it used to in the past. Distributing it is a great feature but unfortunately is no good = in my current setup.</div> <div><br> </div> <div><br> </div> <div>Regards,</div> <div><br> </div> <div>Jeremy </div> <div><br> </div> <br> <p></p> </div> </body> </html> --_000_SL2P216MB0906F39F025F11FBE21AFB72ACA00SL2P216MB0906KORP_--

2 4

回复：回复：Re: ovirt engine HA
by dhy336＠sina.com 08 Apr '18

08 Apr '18

--=-sinamail_alt_14c2cdd577afe05742bb230a8236af4d Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: base64 Content-Disposition: inline CkhpLCBJIGhhdmEgdHdvIG5vZGUsIGFuZCBkZXBsb3kgaG9zdGVkLWVuZ2luZSBieSAjaG9zdGVk LWVuZ2luZSAtLWRlcGxveSwgSSBmaW5kIHR3byBob3N0ZWQtZW5naW5lIHRoYXQgIGkgZGVwbG9p ZWQgaXMgaW5kZXBlbmRlbnQsICBob3cgdG8gbWFrZSBteSBob3N0ZWQtZW5naW5lIGlzIEhBPwot LS0tLSDUrcq808q8/iAtLS0tLQq3orz+yMujukFsZXggSyA8cmlnaHRraWNrdGVjaEBnbWFpbC5j b20+CsrVvP7Iy6O6RkVSTkFORE8gRlJFRElBTkkgPGZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb20+ CrOty83Iy6O6dXNlcnMgPHVzZXJzQG92aXJ0Lm9yZz4K1vfM4qO6UmU6IFtvdmlydC11c2Vyc10g b3ZpcnQgZW5naW5lIEhBCsjVxtqjujIwMTjE6jA01MIwNMjVIDAxteM0MLfWCgpJbiBjYXNlIHlv dSBuZWVkIEhBIGZvciB0aGUgZW5naW5lIHlvdSBuZWVkIHRvIGRlcGxveSBpdCB0byBvdGhlciBo b3N0cyBhbHNvIHRocm91Z2ggdGhlIEdVSS4KCgpPbiBUdWUsIEFwciAzLCAyMDE4IGF0IDQ6NDcg UE0sIEZFUk5BTkRPIEZSRURJQU5JIDxmZXJuYW5kby5mcmVkaWFuaUB1cHguY29tPiB3cm90ZToK SXMgaXQgZW5vdWdoIHRvIGRlcGxveSB0aGUgU2VsZi1Ib3N0ZWQgZW5naW5lIGluIGp1c3Qgb25l IEhvc3Qgb2YgdGhlIGNsdXN0ZXIgb3IgaXMgaXQgbmVjZXNzYXJ5IHRvIHJlcGVhdCB0aGUgcHJv Y2VzcyBpbiBlYWNoIG9mIHRoZSBub2RlcyB0aGF0IG11c3QgYmUgYWJsZSB0byBydW4gaXQgPwoK VGhhbmtzCkZlcm5hbmRvCgoyMDE4LTA0LTAzIDI6MDEgR01ULTAzOjAwIFZpbmNlbnQgUm95ZXIg PHZpbmNlbnRAZXBpY2VuZXJneS5jYT46ClNhbWUgdGhpbmcsIHRoZSBlbmdpbmUgaW4gdGhpcyBj YXNlIGlzICJzZWxmLWhvc3RlZCIsIGFzIGluLCBpdCBydW5zIGluIGEgVk0gaG9zdGVkIG9uIHRo ZSBjbHVzdGVyIHRoYXQgaXQgaXMgbWFuYWdpbmcuICBJIGFtIGEgYmVnaW5uZXIgaGVyZSwgYnV0 IGZyb20gbXkgdW5kZXJzdGFuZGluZywgZWFjaCBub2RlIGlzIGFsd2F5cyBjaGVja2luZyBvbiB0 aGUgaGVhbHRoIG9mIHRoZSBlbmdpbmUgVk0uICBJZiB0aGUgZW5naW5lIGlzIG1pc3NpbmcgKGll LCB0aGUgaG9zdCBydW5uaW5nIGl0IGhhcyBnb25lIGRvd24pLCB0aGVuIGFub3RoZXIgYXZhaWxh YmxlLCBoZWFsdGh5IGhvc3Qgd2lsbCBzcGF3biB1cCB0aGUgZW5naW5lIGFuZCB5b3Ugd2lsbCBy ZWdhaW4gYWNjZXNzLiAKSW4gbXkgZXhwZXJpZW5jZSB0aGlzIGhhcyB3b3JrZWQgdmVyeSByZWxp YWJseS4gIEkgaGF2ZSAyIGhvc3RzLCBib3RoIGFyZSAiYWJsZSIgdG8gcnVuIHRoZSBlbmdpbmUg Vk0uICBJZiBJIHRha2Ugb25lIGhvc3QgZG93biwgSSBhbSBub3QgYWJsZSB0byBsb2FkIHRoZSBl bmdpbmUgR1VJLiAgQnV0IGlmIEkgd2FpdCBhIGZldyBtaW51dGVzLCB0aGVuIEkgcmVnYWluIGFj Y2VzcywgYW5kIHNlZSB0aGF0IHRoZSBlbmdpbmUgaXMgbm93IHJ1bm5pbmcgb24gdGhlIHJlbWFp bmluZyBoZWFsdGh5IGhvc3QuIFZpbmNlbnQgUm95ZXI3NzgtODI1LTEwNTcKClNVU1RBSU5BQkxF IE1PQklMRSBFTkVSR1kgU09MVVRJT05TCgoKDQoKT24gTW9uLCBBcHIgMiwgMjAxOCBhdCA2OjA3 IFBNLCAgPGRoeTMzNkBzaW5hLmNvbT4gd3JvdGU6CndoYXQgZGlmZmVyZW50IGJldHdlZW4gc2Vs Zi1ob3N0ZWQgZW5naW5lIGFuZCAgaG9zdGVkIGVuZ2luZT8gSSBmaW5kIGEgcHJvamVjdCBvdmly dC1ob3N0ZWQtZW5naW5lLWhhICBodHRwczovL2dpdGh1Yi5jb20vb1ZpcnQvb3ZpcnQtaG9zdGVk LWVuZ2luZS1oYSAgLS0tLS0g1K3KvNPKvP4gLS0tLS0Kt6K8/sjLo7pWaW5jZW50IFJveWVyIDx2 aW5jZW50QGVwaWNlbmVyZ3kuY2E+CsrVvP7Iy6O6ZGh5MzM2QHNpbmEuY29tCrOty83Iy6O6dXNl cnMgPHVzZXJzQG92aXJ0Lm9yZz4K1vfM4qO6UmU6IFtvdmlydC11c2Vyc10gb3ZpcnQgZW5naW5l IEhBCsjVxtqjujIwMTjE6jA01MIwM8jVIDA4teM1N7fWCgpJZiB5b3VyIG5vZGUgcnVubmluZyBz ZWxmLWhvc3RlZCBlbmdpbmUgY3Jhc2hlcywgdGhlIGhvc3RlZCBlbmdpbmUgd2lsbCBiZSBzdGFy dGVkIHVwIG9uIGFub3RoZXIgbm9kZS4gSXQganVzdCB0YWtlcyBhIGZldyBtaW51dGVzIGZvciB0 aGlzIGFsbCB0byBoYXBwZW4sIGJ1dCBpdCB3b3JrcyByZWxpYWJseSBpbiBteSBleHBlcmllbmNl LiAKCg0KCk9uIE1vbiwgQXByIDIsIDIwMTggYXQgNTo0MiBQTSwgIDxkaHkzMzZAc2luYS5jb20+ IHdyb3RlOgpIb3cgdG8gc29sdXRlIG92aXJ0IGVuZ2luZSBIQSwgSSBoYXZlIGEgdGhyZWUgbm9k ZSBjbHVzdGVyLCBvbmUgb2YgaXMgZGVwbG95IGVuZ2luZSBhbmQgbm9kZSAsIG90aGVycyBhcmUg bm9kZSwgaWYgbm9kZSB0aGF0IGRlcGxheSBlbmdpbmUgYW5kICBub2RlIGNyYXNoLCBIb3cgdG8g ZW5zdXJlIG15IHNlcnZlciBpcyB1cD8KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX18KDQpVc2VycyBtYWlsaW5nIGxpc3QKDQpVc2Vyc0BvdmlydC5vcmcKDQpo dHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKDQoKCg0KDQoKDQoK X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KDQpVc2VycyBt YWlsaW5nIGxpc3QKDQpVc2Vyc0BvdmlydC5vcmcKDQpodHRwOi8vbGlzdHMub3ZpcnQub3JnL21h aWxtYW4vbGlzdGluZm8vdXNlcnMKDQoKCg0KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fCg0KVXNlcnMgbWFpbGluZyBsaXN0Cg0KVXNlcnNAb3ZpcnQub3Jn Cg0KaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCg0KCgoNCg0K X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVXNlcnMgbWFp bGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZwpodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v bGlzdGluZm8vdXNlcnMKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KVXNlcnMgbWFpbGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZwpodHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMK --=-sinamail_alt_14c2cdd577afe05742bb230a8236af4d Content-Type: text/html; charset=GBK Content-Transfer-Encoding: base64 Content-Disposition: inline PGRpdiBpZD0ib3JpZ2JvZHkiPjxkaXYgc3R5bGU9ImJhY2tncm91bmQ6ICNmMmYyZjI7Ij48YnI+ PC9kaXY+PGRpdj5IaSwgSSBoYXZhIHR3byBub2RlLCBhbmQgZGVwbG95IGhvc3RlZC1lbmdpbmUg YnkgI2hvc3RlZC1lbmdpbmUgLS1kZXBsb3ksPC9kaXY+PGRpdj4mbmJzcDtJIGZpbmQgdHdvIGhv c3RlZC1lbmdpbmUgdGhhdCAmbmJzcDtpIGRlcGxvaWVkIGlzIGluZGVwZW5kZW50LCAmbmJzcDto b3cgdG8gbWFrZSBteSBob3N0ZWQtZW5naW5lIGlzIEhBPzwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxk aXYgaWQ9Im9yaWdib2R5Ij48ZGl2IHN0eWxlPSJiYWNrZ3JvdW5kOiAjZjJmMmYyOyI+LS0tLS0g 1K3KvNPKvP4gLS0tLS08YnI+t6K8/sjLo7pBbGV4IEsgJmx0O3JpZ2h0a2lja3RlY2hAZ21haWwu Y29tJmd0Ozxicj7K1bz+yMujukZFUk5BTkRPIEZSRURJQU5JICZsdDtmZXJuYW5kby5mcmVkaWFu aUB1cHguY29tJmd0Ozxicj6zrcvNyMujunVzZXJzICZsdDt1c2Vyc0BvdmlydC5vcmcmZ3Q7PGJy Ptb3zOKjulJlOiBbb3ZpcnQtdXNlcnNdIG92aXJ0IGVuZ2luZSBIQTxicj7I1cbao7oyMDE4xOow NNTCMDTI1SAwMbXjNDC31jxicj48L2Rpdj48YnI+PGRpdiBkaXI9Imx0ciI+SW4gY2FzZSB5b3Ug bmVlZCBIQSBmb3IgdGhlIGVuZ2luZSB5b3UgbmVlZCB0byBkZXBsb3kgaXQgdG8gb3RoZXIgaG9z dHMgYWxzbyB0aHJvdWdoIHRoZSBHVUkuPGJyPjxicj48L2Rpdj48ZGl2IGNsYXNzPSIiPjxicj48 ZGl2IGNsYXNzPSIiPk9uIFR1ZSwgQXByIDMsIDIwMTggYXQgNDo0NyBQTSwgRkVSTkFORE8gRlJF RElBTkkgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0ibWFpbHRv OmZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb20iPmZlcm5hbmRvLmZyZWRpYW5pQHVweC5jb208L2E+ Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSIiIHN0eWxlPSJtYXJnaW46 MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48 ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXY+SXMgaXQgZW5vdWdoIHRvIGRlcGxveSB0aGUgU2VsZi1I b3N0ZWQgZW5naW5lIGluIGp1c3Qgb25lIEhvc3Qgb2YgdGhlIGNsdXN0ZXIgb3IgaXMgaXQgbmVj ZXNzYXJ5IHRvIHJlcGVhdCB0aGUgcHJvY2VzcyBpbiBlYWNoIG9mIHRoZSBub2RlcyB0aGF0IG11 c3QgYmUgYWJsZSB0byBydW4gaXQgPzxicj48YnI+PC9kaXY+VGhhbmtzPHNwYW4gY2xhc3M9IiI+ PGZvbnQgY29sb3I9IiM4ODg4ODgiPjxicj48L2ZvbnQ+PC9zcGFuPjwvZGl2PjxzcGFuIGNsYXNz PSIiPjxmb250IGNvbG9yPSIjODg4ODg4Ij5GZXJuYW5kbzxicj48L2ZvbnQ+PC9zcGFuPjwvZGl2 PjxkaXYgY2xhc3M9IiI+PGRpdiBjbGFzcz0iIj48ZGl2IGNsYXNzPSIiPjxicj48ZGl2IGNsYXNz PSIiPjIwMTgtMDQtMDMgMjowMSBHTVQtMDM6MDAgVmluY2VudCBSb3llciA8c3BhbiBkaXI9Imx0 ciI+Jmx0OzxhIHRhcmdldD0iX2JsYW5rIiBocmVmPSJtYWlsdG86dmluY2VudEBlcGljZW5lcmd5 LmNhIj52aW5jZW50QGVwaWNlbmVyZ3kuY2E8L2E+Jmd0Ozwvc3Bhbj46PGJyPjxibG9ja3F1b3Rl IGNsYXNzPSIiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBz b2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj5TYW1lIHRoaW5nLCB0aGUgZW5n aW5lIGluIHRoaXMgY2FzZSBpcyAic2VsZi1ob3N0ZWQiLCBhcyBpbiwgaXQgcnVucyBpbiBhIFZN IGhvc3RlZCBvbiB0aGUgY2x1c3RlciB0aGF0IGl0IGlzIG1hbmFnaW5nLiZuYnNwOyBJIGFtIGEg YmVnaW5uZXIgaGVyZSwgYnV0IGZyb20gbXkgdW5kZXJzdGFuZGluZywgZWFjaCBub2RlIGlzIGFs d2F5cyBjaGVja2luZyBvbiB0aGUgaGVhbHRoIG9mIHRoZSBlbmdpbmUgVk0uJm5ic3A7IElmIHRo ZSBlbmdpbmUgaXMgbWlzc2luZyAoaWUsIHRoZSBob3N0IHJ1bm5pbmcgaXQgaGFzIGdvbmUgZG93 biksIHRoZW4gYW5vdGhlciBhdmFpbGFibGUsIGhlYWx0aHkgaG9zdCB3aWxsIHNwYXduIHVwIHRo ZSBlbmdpbmUgYW5kIHlvdSB3aWxsIHJlZ2FpbiBhY2Nlc3MuJm5ic3A7PGRpdj48YnI+PC9kaXY+ PGRpdj5JbiBteSBleHBlcmllbmNlIHRoaXMgaGFzIHdvcmtlZCB2ZXJ5IHJlbGlhYmx5LiZuYnNw OyBJIGhhdmUgMiBob3N0cywgYm90aCBhcmUgImFibGUiIHRvIHJ1biB0aGUgZW5naW5lIFZNLiZu YnNwOyBJZiBJIHRha2Ugb25lIGhvc3QgZG93biwgSSBhbSBub3QgYWJsZSB0byBsb2FkIHRoZSBl bmdpbmUgR1VJLiZuYnNwOyBCdXQgaWYgSSB3YWl0IGEgZmV3IG1pbnV0ZXMsIHRoZW4gSSByZWdh aW4gYWNjZXNzLCBhbmQgc2VlIHRoYXQgdGhlIGVuZ2luZSBpcyBub3cgcnVubmluZyBvbiB0aGUg cmVtYWluaW5nIGhlYWx0aHkgaG9zdC4mbmJzcDs8L2Rpdj48L2Rpdj48ZGl2IGNsYXNzPSIiPjxi ciBjbGVhcj0iYWxsIj48ZGl2PjxkaXYgY2xhc3M9IiIgZGF0YS1zbWFydG1haWw9ImdtYWlsX3Np Z25hdHVyZSI+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXYgZGly PSJsdHIiPjxkaXY+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXYg ZGlyPSJsdHIiPjxkaXY+PGRpdj48ZGl2IGRpcj0ibHRyIiBzdHlsZT0iZm9udC1zaXplOjEyLjhw eCI+PGZvbnQgc2l6ZT0iNCIgZmFjZT0iYXJpYWwgbmFycm93LCBzYW5zLXNlcmlmIj48Yj5WaW5j ZW50IFJveWVyPC9iPjwvZm9udD48L2Rpdj48ZGl2IGRpcj0ibHRyIj48ZGl2IHN0eWxlPSJmb250 LXNpemU6MTIuOHB4Ij48Zm9udCBzaXplPSI0IiBmYWNlPSJhcmlhbCBuYXJyb3csIHNhbnMtc2Vy aWYiPjxiPjxhIHRhcmdldD0iX2JsYW5rIiBocmVmPSJ0ZWw6KDc3OCklMjA4MjUtMTA1NyIgdmFs dWU9IisxNzc4ODI1MTA1NyI+Nzc4LTgyNS0xMDU3PC9hPjwvYj48L2ZvbnQ+PC9kaXY+PGRpdiBz dHlsZT0iZm9udC1zaXplOjEyLjhweCI+PGZvbnQgc2l6ZT0iNCIgZmFjZT0iYXJpYWwgbmFycm93 LCBzYW5zLXNlcmlmIj48Yj48YnI+PC9iPjwvZm9udD48L2Rpdj48ZGl2IHN0eWxlPSJmb250LXNp emU6MTIuOHB4Ij48YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0iaHR0cDovL3d3dy5lcGljZW5lcmd5 LmNhLyI+PGltZyBzcmM9Imh0dHBzOi8vbGgzLmdvb2dsZXVzZXJjb250ZW50LmNvbS9lekQ5Snhl TjY3bFRzY3NlbEhqVzRaRXF2UVJqX2lfVF9LeV9zX19Ha1IwQXRhbkcxbks1MkNEMVJkaG15emJr N1hSUGYtUmhobnIyLVNMUG1odGQwSjh4Vk1wMHhPaVIyMGViUVZiNmpVMnh3SlBLTjRsZVBScTVH MnA5M3licUE0U25xakNXQ3ZFc05sSzM3YTlGdFN0ZEZjVEp5SW9TMmFIS2lMNjlMM2JaZzFnN21F QU1yN0lIWFZZdkhncTNNWDlyTmFxSVF4OVQzdVB1SUhFRElmeHJncENyc0JvRWFtZ2FSSGo1aGNw RW9ibm9ncUNxcGExc1JjS3k5MjdUU2VSNUJiVkxNMS13bjBoLTRMdkU4Q01xajMwanBRVUx4Wjl1 WEVnX1M2c1IzWmVaanRra1J2Z0s0bF93aWl2V2x5Vl9TakNEUTVRUkVHM2prdUNqWEMwNGkxTk5t emlHWlpOV1lzYW9oRTZraHJONEpHVnR2YVNPaVMyd29ZSk56M0d0MEV1Y3hKcDNlVUJZc3B6VGw5 aVppYTdiTHZGRUdJT0R2bWZqUlZaSm1uQ3JGdWNKeEdrQXRXbl9FVUlieEpYUzVOV0FGbnZCb2l4 UVA1NjNGYnU3ZEZLTDc3bERjYjRQSHdybkpWanFjZ1UwT1RlaU1CbE12RXQ5N1lnTXBNdmRpN3J4 OUw0RGVFYnJJR1Vvdmd4NFFMZlJSb0ZfOXUybXhGdThGSHVwQ3hmb2NTeUIzdnM4Z3FPTHZWZHg0 MnF3aUxOMHNHTEFnOEszR192RndWTmRkWFFfeWIzY0dlQ0VBT1RuazdqZmlQNE80anNQSVcyUlJi bktLTC11c0wxMVRwN2dreGJnUm91TE9veXFwYmlzakE0dlJXRGV5bGoxeURHeUJnPXc0MDAtaDM4 LW5vIiB3aWR0aD0iMjAwIiBoZWlnaHQ9IjE4Ij48YnI+PC9hPjwvZGl2PjxkaXY+PGI+PGZvbnQg c2l6ZT0iMiIgZmFjZT0iYXJpYWwgbmFycm93LCBzYW5zLXNlcmlmIj5TVVNUQUlOQUJMRSBNT0JJ TEUmbmJzcDs8Zm9udCBjb2xvcj0iIzZhYTg0ZiI+RU5FUkdZJm5ic3A7PC9mb250PlNPTFVUSU9O UzwvZm9udD48L2I+PC9kaXY+PC9kaXY+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+PGRp dj48ZGl2Pjxicj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rp dj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48 ZGl2PjxkaXYgY2xhc3M9IiI+DQo8YnI+PGRpdiBjbGFzcz0iIj5PbiBNb24sIEFwciAyLCAyMDE4 IGF0IDY6MDcgUE0sICA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIHRhcmdldD0iX2JsYW5rIiBocmVm PSJtYWlsdG86ZGh5MzM2QHNpbmEuY29tIj5kaHkzMzZAc2luYS5jb208L2E+Jmd0Ozwvc3Bhbj4g d3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSIiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDti b3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48Zm9udCBzaXplPSI0 Ij48Zm9udCBmYWNlPSJBcmlhbCwgTWljcm9zb2Z0IFlhSGVpLCDOosjt0cW62iwgy87M5SwgTWFs Z3VuIEdvdGhpYywgTWVpcnlvLCBzYW5zLXNlcmlmIj48c3BhbiBzdHlsZT0iYmFja2dyb3VuZC1j b2xvcjpyZ2IoMjQ5LDI1MSwyNTIpIj53aGF0IGRpZmZlcmVudCBiZXR3ZWVuJm5ic3A7PC9zcGFu PjwvZm9udD5zZWxmLWhvc3RlZCBlbmdpbmUgYW5kICZuYnNwO2hvc3RlZCBlbmdpbmU/IEkgZmlu ZCBhIHByb2plY3QmbmJzcDs8L2ZvbnQ+PGgxIGNsYXNzPSIiIHN0eWxlPSJib3gtc2l6aW5nOmJv cmRlci1ib3g7bWFyZ2luOjBweDtsaW5lLWhlaWdodDoyNnB4O21heC13aWR0aDo2MzVweDtwYWRk aW5nLWxlZnQ6MThweDtjb2xvcjpyZ2IoODgsOTYsMTA1KTtmb250LWZhbWlseTotYXBwbGUtc3lz dGVtLEJsaW5rTWFjU3lzdGVtRm9udCwmcXVvdDtTZWdvZSBVSSZxdW90OyxIZWx2ZXRpY2EsQXJp YWwsc2Fucy1zZXJpZiwmcXVvdDtBcHBsZSBDb2xvciBFbW9qaSZxdW90OywmcXVvdDtTZWdvZSBV SSBFbW9qaSZxdW90OywmcXVvdDtTZWdvZSBVSSBTeW1ib2wmcXVvdDs7YmFja2dyb3VuZC1jb2xv cjpyZ2IoMjUwLDI1MSwyNTIpIj48c3BhbiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2Zv bnQtd2VpZ2h0Om5vcm1hbCI+PGZvbnQgc2l6ZT0iNCI+PGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9 Imh0dHBzOi8vZ2l0aHViLmNvbS9vVmlydC9vdmlydC1ob3N0ZWQtZW5naW5lLWhhIiBzdHlsZT0i Ym94LXNpemluZzpib3JkZXItYm94O2JhY2tncm91bmQtY29sb3I6dHJhbnNwYXJlbnQ7Y29sb3I6 cmdiKDMsMTAyLDIxNCk7b3V0bGluZS13aWR0aDowcHgiPm92aXJ0LWhvc3RlZC1lbmdpbmUtaGE8 L2E+Jm5ic3A7Jm5ic3A7PGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Imh0dHBzOi8vZ2l0aHViLmNv bS9vVmlydC9vdmlydC1ob3N0ZWQtZW5naW5lLWhhIj5odHRwczo8d2JyPi8vZ2l0aHViLmNvbS9v VmlydC9vdmlydC1ob3N0ZTx3YnI+ZC1lbmdpbmUtaGE8L2E+ICZuYnNwOzwvZm9udD48L3NwYW4+ PC9oMT48ZGl2IGlkPSJtXy0yMDg1MDkxNjQ0MDIxNTU2NTkwbV8yMzU0OTMxMDMzNzI4MDI3NDkw bV8zMDA3OTMxODM5NzY0OTc0NzQ3b3JpZ2JvZHkiPjxkaXYgc3R5bGU9ImJhY2tncm91bmQ6I2Yy ZjJmMiI+LS0tLS0g1K3KvNPKvP4gLS0tLS08YnI+t6K8/sjLo7pWaW5jZW50IFJveWVyICZsdDs8 YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0ibWFpbHRvOnZpbmNlbnRAZXBpY2VuZXJneS5jYSI+dmlu Y2VudEBlcGljZW5lcmd5LmNhPC9hPiZndDs8YnI+ytW8/sjLo7o8YSB0YXJnZXQ9Il9ibGFuayIg aHJlZj0ibWFpbHRvOmRoeTMzNkBzaW5hLmNvbSI+ZGh5MzM2QHNpbmEuY29tPC9hPjxicj6zrcvN yMujunVzZXJzICZsdDs8YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0ibWFpbHRvOnVzZXJzQG92aXJ0 Lm9yZyI+dXNlcnNAb3ZpcnQub3JnPC9hPiZndDs8YnI+1vfM4qO6UmU6IFtvdmlydC11c2Vyc10g b3ZpcnQgZW5naW5lIEhBPGJyPsjVxtqjujIwMTjE6jA01MIwM8jVIDA4teM1N7fWPGJyPjwvZGl2 PjxkaXY+PGRpdiBjbGFzcz0iIj48YnI+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2PjxkaXYgZGF0 YS1zbWFydG1haWw9ImdtYWlsX3NpZ25hdHVyZSI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0 ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9 Imx0ciI+PGRpdj48ZGl2PjxkaXY+SWYgeW91ciBub2RlIHJ1bm5pbmcgc2VsZi1ob3N0ZWQgZW5n aW5lIGNyYXNoZXMsIHRoZSBob3N0ZWQgZW5naW5lIHdpbGwgYmUgc3RhcnRlZCB1cCBvbiBhbm90 aGVyIG5vZGUuIEl0IGp1c3QgdGFrZXMgYSBmZXcgbWludXRlcyBmb3IgdGhpcyBhbGwgdG8gaGFw cGVuLCBidXQgaXQgd29ya3MgcmVsaWFibHkgaW4gbXkgZXhwZXJpZW5jZS4mbmJzcDs8L2Rpdj48 L2Rpdj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rp dj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj4NCjxicj48ZGl2Pk9uIE1vbiwgQXByIDIs IDIwMTggYXQgNTo0MiBQTSwgIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgdGFyZ2V0PSJfYmxhbmsi IGhyZWY9Im1haWx0bzpkaHkzMzZAc2luYS5jb20iPmRoeTMzNkBzaW5hLmNvbTwvYT4mZ3Q7PC9z cGFuPiB3cm90ZTo8YnI+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRl ci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPkhvdyB0byBzb2x1dGUgb3Zp cnQgZW5naW5lIEhBLCBJIGhhdmUgYSB0aHJlZSBub2RlIGNsdXN0ZXIsIG9uZSBvZiBpcyBkZXBs b3kgZW5naW5lIGFuZCBub2RlICwgb3RoZXJzIGFyZSBub2RlLCBpZiBub2RlIHRoYXQgZGVwbGF5 IGVuZ2luZSBhbmQgJm5ic3A7bm9kZSBjcmFzaCwgSG93IHRvIGVuc3VyZSBteSBzZXJ2ZXIgaXMg dXA/PGJyPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19f X188YnI+DQpVc2VycyBtYWlsaW5nIGxpc3Q8YnI+DQo8YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0i bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3ZpcnQub3JnPC9hPjxicj4NCjxhIHRhcmdl dD0iX2JsYW5rIiBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8v dXNlcnMiIHJlbD0ibm9yZWZlcnJlciI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdi cj4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPg0KPGJyPjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9k aXY+PC9kaXY+DQoNCjwvZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9k aXY+PC9kaXY+PC9kaXY+DQo8YnI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPHdicj5f X19fX19fX19fX19fX19fXzxicj4NClVzZXJzIG1haWxpbmcgbGlzdDxicj4NCjxhIHRhcmdldD0i X2JsYW5rIiBocmVmPSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIj5Vc2Vyc0BvdmlydC5vcmc8L2E+ PGJyPg0KPGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFp bG1hbi9saXN0aW5mby91c2VycyIgcmVsPSJub3JlZmVycmVyIj5odHRwOi8vbGlzdHMub3ZpcnQu b3JnL21haWxtYW48d2JyPi9saXN0aW5mby91c2VyczwvYT48YnI+DQo8YnI+PC9ibG9ja3F1b3Rl PjwvZGl2Pjxicj48L2Rpdj4NCjwvZGl2PjwvZGl2Pjxicj5fX19fX19fX19fX19fX19fX19fX19f X19fX19fX188d2JyPl9fX19fX19fX19fX19fX19fPGJyPg0KVXNlcnMgbWFpbGluZyBsaXN0PGJy Pg0KPGEgdGFyZ2V0PSJfYmxhbmsiIGhyZWY9Im1haWx0bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJz QG92aXJ0Lm9yZzwvYT48YnI+DQo8YSB0YXJnZXQ9Il9ibGFuayIgaHJlZj0iaHR0cDovL2xpc3Rz Lm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIiByZWw9Im5vcmVmZXJyZXIiPmh0dHA6 Ly9saXN0cy5vdmlydC5vcmcvPHdicj5tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPC9hPjxicj4NCjxi cj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2Pg0KDQpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXzxicj5Vc2VycyBtYWlsaW5nIGxpc3Q8YnI+VXNlcnNA b3ZpcnQub3JnPGJyPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vy czxicj48L2Rpdj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f Xzxicj5Vc2VycyBtYWlsaW5nIGxpc3Q8YnI+VXNlcnNAb3ZpcnQub3JnPGJyPmh0dHA6Ly9saXN0 cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vyczxicj48L2Rpdj4= --=-sinamail_alt_14c2cdd577afe05742bb230a8236af4d--

2 1

Is compatibility level change required for upgrading from 4.0 to 4.2?
by Luca 'remix_tj' Lorenzetto 08 Apr '18

08 Apr '18

Hello, we're planning an upgrade of an old 4.0 setup to 4.2, going through 4.1. What we found out is that when upgrading from major to major, cluster and datacenter compatibility upgrade has to be done at the end of the upgrade. This means that we also require to restart our VMs for adapting the compatibility level. Do we require to upgrade the compatibility level to 4.1 before starting the upgrade to 4.2? We have more than 400 vms and only few of them can be restarted at our convenience. -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca(a)gmail.com>

4 5

Hardware critique
by Jayme 06 Apr '18

06 Apr '18

I'm spec'ing hardware for a 3-node oVirt build (on somewhat of a budget). I plan to do 20-30 Linux VMs most of them very light weight + a couple of heavier hitting web and DB servers with frequent rsync backups. Some have a lot of small files from large github repos etc. 3X of the following: Dell PowerEdge R720 2x 2.9 GHz 8 Core E5-2690 (SR0L0) 256GB RAM PERC H710 2x10GB Nic Boot/OS will likely be two cheaper small sata/ssd in raid 1. Gluster bricks comprised of 4x2TB WD Gold 7200RPM SATA HDDs in RAID 10 per server. Using a replica 3 setup (and I'm thinking right now with no arbiter for extra redundancy, although I'm not sure what the performance hit may be as a result). Will this allow for two host failure or just one? I've been really struggling with storage choices, it seems very difficult to predict the performance of glusterFS due to the variance in hardware (everyone is using something different). I'm not sure if the performance will be adequate enough for my needs. I will be using an all ready existing Netgear XS716T 10GB switch for Gluster storage network. In addition I plan to build another simple glusterFS storage server that I can use to georeplicate the gluster volume to for DR purposes and use existing hardware to build an independent standby oVirt host that is able to start up a few high priority VMs from the georeplicated glusterFS volume if for some reason the primary oVirt cluster/glusterFS volume ever failed. I would love to hear any advice or critiques on this plan. Thanks!

7 13

ISO uploading from GUI/REST with user permissions
by Lloyd Kamara 06 Apr '18

06 Apr '18

Dear Sir/Madam, The ability to upload ISOs through the web interface and boot VMs from them is a welcome addition in oVirt release 4.2.2. I am grateful to the people behind the implementation of this. Consider a scenario in which you wish to allow *end-users* to upload ISOs to one or more Data Domains. The users can then use the uploaded ISOs to boot their VMs. Is it possible to grant a user permission to upload ISOs through the web interface? I tried to to this under oVirt release 4.2.2 by doing the following: - adding the 'SuperUser' role to a target user for a specific Data Domain, which enables the user to log onto the Administration Portal. - adding the 'DiskCreator' role to the same target user for the same Data Domain, which, I would hope, would allow the user to both create disks and upload ISOs within that Data Domain. Disk creation in the Data Domain for the target user works as expected; ISO upload does not. A dialog appears with the message: 'Operation Canceled Error while executing action: User is not authorized to perform this action.' Here is the message that appears in /var/log/ovirt-engine/engine.log when an attempt at uploading an ISO is made by the target user: INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-40) [5b3fef06-49c8-4c34-81a3-a20fa691709a] No permission found for user 'a9fde4c3-97a3-4494-84f8-08041a16710c' or one of the groups he is member of, when running action 'TransferImageStatus', Required permissions are: Action type: 'USER' Action group: 'CREATE_DISK' Object type: 'System' Object ID: 'aaa00000-0000-0000-0000-123456789aaa'. If one assigns the DiskCreator role System permission for the target user then that user can upload ISOs without problem. Unfortunately, the user can upload ISOs - and create disks - in *all* data domains. To re-iterate, is it possible to grant an end-user permission to upload ISOs to specific data domains through the web interface without granting an all-encompassing System permission? Best wishes, Lloyd Kamara References: [The first two are included insofar as they concern ISO upload via web] https://bugzilla.redhat.com/show_bug.cgi?id=1530730 https://bugzilla.redhat.com/show_bug.cgi?id=1536826 [This one is included because I wonder if the testing requests includes the ability for users to upload ISOs via the web GUI, not just attach existing ISOs in data domains to VMs] https://bugzilla.redhat.com/show_bug.cgi?id=1058798

2 2

how do you backup VM
by Peter Hudec 06 Apr '18

06 Apr '18

Hi, one general question. See the $SUBJ. I have found https://github.com/openbacchus/bacchus, that good as start point, but still missing some features. i was thinking to contribute here, but first I want to known another solutions. regards Peter -- *Peter Hudec* Infraštruktúrny architekt phudec(a)cnc.sk <mailto:phudec@cnc.sk> *CNC, a.s.* Borská 6, 841 04 Bratislava Recepcia: +421 2 35 000 100 Mobil:+421 905 997 203 *www.cnc.sk* <http:///www.cnc.sk>

1 0

Greetings oVirt Users
by Clint Boggio 06 Apr '18

06 Apr '18

Environment Rundown: OVirt 4.2 6 CentOS 7.4 Compute Nodes Intel Xeon 1 CentOS 7.4 Dedicated Engine Node Intel Xeon 1 Datacenter 1 Storage Domain 1 Cluster 10Gig-E iSCSI Storage 10Gig-E NFS Export Domain 20 VM’s of various OS’s and uses The current cluster is using the Nehalem architecture. I’ve got the deploy two new VMs that the current system will not allow me to configure with the Nehalem based cluster, so I’ve got to bump up the architecture of the cluster to accommodate them. Before i shut down all the current VMs to upgrade the cluster, I have some questions about the effect this is going to have on the environment. 1. Will all of the current VM’s use the legacy processor architecture or will I have to change them ? 2. Can I elevate the cluster processor functionality higher than the underlying hardware architecture ? 3. In regards to the new cluster processor, will all of the processor architectures below the one I choose be an option for the existing and future VMs ? I apologize for the long post and I hope that I haven’t left out any vital information.

2 1

Why RAW images when using GlusterFS?
by Nicolas Ecarnot 05 Apr '18

05 Apr '18

Hello, Amongst others, I have one 3.6 DC working very well since years and all based on GlusterFS. When having a close look (qemu-img info) on the images, I see their format is all RAW and not QCOW2. I never noticed or bothered before, but I'm wondering : - is it by design? - it is something we can change (I'd prefer qcow2) - it there some limitations? And finally, I have the same questions about NFS storage domains. Thank you. -- Nicolas ECARNOT

2 1

Updates to oVirt 4.2.2
by Sandro Bonazzola 05 Apr '18

05 Apr '18

Hi, the oVirt team released today April 5th an update to oVirt 4.2.2 including the following packages: - ovirt-hosted-engine-ha-2.2.10 - ovirt-hosted-engine-setup-2.2.16 - cockpit-ovirt-0.11.20-1 - ovirt-release42-4.2.2-3 Addressing the following issues: - [BZ 1560666 <https://bugzilla.redhat.com/1560666>] - Hosted Engine VM (deployed in the past) fails to reboot with 'libvirtError: internal error: failed to format device alias for PTY retrieval' due to an error in console device in libvirt XML generated by the engine - [BZ 1560551 <https://bugzilla.redhat.com/1560551>] - The user is asked twice to enter engine VM FQDN - [BZ 1560655 <https://bugzilla.redhat.com/1560655>] - Node 0 flow is consuming the portal IP used for the discovery. - [BZ 1562349 <https://bugzilla.redhat.com/1562349>] - Upgrade appliance failed with 'Environment customization': 'OVEHOSTED_STORAGE/spUUID' - [BZ 1563664 <https://bugzilla.redhat.com/1563664>] - Ansible deployments fails on NFS with root_squash option - [BZ 1562011 <https://bugzilla.redhat.com/1562011>] - Ansible: Deploy HE failed with FC storage via cockpit. - Switch to signed RPMs yum repository for CentOS Virt SIG on ppc64le An update for oVirt Node is already available. An updated oVirt Node ISO is being composed right now and will be available for download by tomorrow morning. -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>

1 0

How to subscribe to Content Delivery Network in oVirt 4.2.2 Cockpit
by Canglin 04 Apr '18

04 Apr '18

This is a multi-part message in MIME format. --------------C2F640B89AA37D2F950B12D8 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit In the documentation - chapter 6 "Installing oVirt Node" version 4.2.2. Step 13. Once the installation is complete, log in to the Cockpit user interface at https:///HostFQDNorIP/:9090 to subscribe the host to the Content Delivery Network. Click *Tools* > *Subscriptions* > *Register System* and enter your Customer Portal username and password. The system automatically subscribes to the *oVirt Node* entitlement. I am having a hard time finding this in the cockpit. Any assistance in explaining the benefit of this step and where to find would be appreciated. Thanks --------------C2F640B89AA37D2F950B12D8 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>In the documentation - chapter 6 "Installing oVirt Node" version 4.2.2.</p> <p>Step 13.</p> <p>Once the installation is complete, log in to the Cockpit user interface at <a class="moz-txt-link-freetext" href="https://">https://</a><em>HostFQDNorIP</em>:9090 to subscribe the host to the Content Delivery Network. Click <strong>Tools</strong> > <strong>Subscriptions</strong> > <strong>Register System</strong> and enter your Customer Portal username and password. The system automatically subscribes to the <strong>oVirt Node</strong> entitlement.</p> <p>I am having a hard time finding this in the cockpit. Any assistance in explaining the benefit of this step and where to find would be appreciated. Thanks</p> <p><br> </p> </body> </html> --------------C2F640B89AA37D2F950B12D8--

1 0