From moacirferreira at hotmail.com Tue Aug 8 11:37:57 2017 Content-Type: multipart/mixed; boundary="===============3670569142046871846==" MIME-Version: 1.0 From: Moacir Ferreira To: users at ovirt.org Subject: Re: [ovirt-users] Good practices Date: Tue, 08 Aug 2017 11:37:54 +0000 Message-ID: In-Reply-To: mailman.4593.1502187832.1857.users@ovirt.org --===============3670569142046871846== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_DB6P190MB0280AD888032AB4A6232D576C88A0DB6P190MB0280EURP_ Content-Type: text/plain; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable Ok, the 40Gb NIC that I got were for free. But anyway, if you were working = =3D with 6 HDD + 1 SSD per server, then you get 21 disks on your cluster. As da= =3D ta in a JBOD will be built all over the network, then it can be really inte= =3D nsive especially depending on the number of replicas you choose for your ne= =3D eds. Also, when moving a VM alive you must transfer the memory contents of = =3D a VM to another node (just think about moving a VM with 32GB RAM). All toge= =3D ther, it can be a quite large chunk of data moving over the network all the= =3D time. While 40Gb NIC is not a "must", I think it is more affordable as it = =3D cost much less then a good disk controller. But my confusion is that, as said by other fellows, the best "performance m= =3D odel" is when you use a hardware RAIDed brick (i.e.: 5 or 6) to assemble yo= =3D ur GlusterFS. In this case, as I would have to buy a good controller but ha= =3D ve less network traffic, to lower the cost I would then use a separate netw= =3D ork made of 10Gb NICs plus the controller. Moacir > > > Le 8 ao?t 2017 ? 04:08, FERNANDO FREDIANI = a > ?crit : > > > Even if you have a Hardware RAID Controller with Writeback cache you > will have a significant performance penalty and may not fully use all the > resources you mentioned you have. > > > > Nope again,from my experience with HP Smart Array and write back cache, > write, that goes in the cache, are even faster that read that must goes t= =3D o > the disks. of course if the write are too fast and to big, they will over > overflow the cache. But on todays controller they are multi-gigabyte cach= =3D e, > you must write a lot to fill them. And if you can afford 40Gb card, you c= =3D an > afford decent controller. > The last sentence raises an excellent point: balance your resources. Don't spend a fortune on one component while another will end up being your bottleneck. Storage is usually the slowest link in the chain. I personally believe that spending the money on NVMe drives makes more sense than 40Gb (except [1], which is suspiciously cheap!) Y. [1] http://a.co/4hsCTqG --_000_DB6P190MB0280AD888032AB4A6232D576C88A0DB6P190MB0280EURP_ Content-Type: text/html; charset=3D"iso-8859-1" Content-Transfer-Encoding: quoted-printable

Ok, the 40Gb NIC that I got were for free. But anyway, if you were worki= =3D ng with 6 HDD + 1 SSD per server, then you get 21 disks on your cluster= =3D . As data in a JBOD will be built all over the network, then it can be real= =3D ly intensive especially depending on the number of replicas you choose for your needs. Also, when moving a VM a= =3D live you must transfer the memory contents of a VM to another node (just th= =3D ink about moving a VM with 32GB RAM). All together, it can be a quite large= =3D chunk of data moving over the network all the time. While 40Gb NIC is not a "must", I think it is more= =3D affordable as it cost much less then a good disk controller.

But my confusion is that, as said by other fellows, the best "perfo= =3D rmance model" is when you use a hardware RAIDed brick (i.e.: 5 or 6) t= =3D o assemble your GlusterFS. In this case, as I would have to buy a good cont= =3D roller but have less network traffic, to lower the cost I would then use a separate network made of 10Gb NICs plus the co= =3D ntroller.

Moacir

>
> > Le 8 ao?t 2017 ? 04:08, FERNANDO FREDIANI <fernando.frediani(a= )u=3D px.com> a
> ?crit :
>
> > Even if you have a Hardware RAID Controller with Writeback cache = =3D you
> will have a significant performance penalty and may not fully use all = =3D the
> resources you mentioned you have.
> >
>
> Nope again,from my experience with HP Smart Array and write back cache= =3D ,
> write, that goes in the cache, are even faster that read that must goe= =3D s to
> the disks. of course if the write are too fast and to big, they will o= =3D ver
> overflow the cache. But on todays controller they are multi-gigabyte c= =3D ache,
> you must write a lot to fill them. And if you can afford 40Gb card, yo= =3D u can
> afford decent controller.
>

The last sentence raises an excellent point: balance your resources. Don't<= =3D br> spend a fortune on one component while another will end up being your
bottleneck.
Storage is usually the slowest link in the chain. I personally believe that= =3D
spending the money on NVMe drives makes more sense than 40Gb (except [1], which is suspiciously cheap!)

Y.
[1] http://a.co/4hsCTqG

--_000_DB6P190MB0280AD888032AB4A6232D576C88A0DB6P190MB0280EURP_-- --===============3670569142046871846== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0RCNlAxOTBNQjAyODBBRDg4ODAzMkFCNEE2MjMyRDU3NkM4OEEwREI2UDE5ME1CMDI4 MEVVUlBfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0iaXNvLTg4NTktMSIKQ29u dGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVvdGVkLXByaW50YWJsZQoKT2ssIHRoZSA0MEdiIE5J QyB0aGF0IEkgZ290IHdlcmUgZm9yIGZyZWUuIEJ1dCBhbnl3YXksIGlmIHlvdSB3ZXJlIHdvcmtp bmcgPQp3aXRoIDYgSEREICsgMSBTU0QgcGVyIHNlcnZlciwgdGhlbiB5b3UgZ2V0IDIxIGRpc2tz IG9uIHlvdXIgY2x1c3Rlci4gQXMgZGE9CnRhIGluIGEgSkJPRCB3aWxsIGJlIGJ1aWx0IGFsbCBv dmVyIHRoZSBuZXR3b3JrLCB0aGVuIGl0IGNhbiBiZSByZWFsbHkgaW50ZT0KbnNpdmUgZXNwZWNp YWxseSBkZXBlbmRpbmcgb24gdGhlIG51bWJlciBvZiByZXBsaWNhcyB5b3UgY2hvb3NlIGZvciB5 b3VyIG5lPQplZHMuIEFsc28sIHdoZW4gbW92aW5nIGEgVk0gYWxpdmUgeW91IG11c3QgdHJhbnNm ZXIgdGhlIG1lbW9yeSBjb250ZW50cyBvZiA9CmEgVk0gdG8gYW5vdGhlciBub2RlIChqdXN0IHRo aW5rIGFib3V0IG1vdmluZyBhIFZNIHdpdGggMzJHQiBSQU0pLiBBbGwgdG9nZT0KdGhlciwgaXQg Y2FuIGJlIGEgcXVpdGUgbGFyZ2UgY2h1bmsgb2YgZGF0YSBtb3Zpbmcgb3ZlciB0aGUgbmV0d29y ayBhbGwgdGhlPQogdGltZS4gV2hpbGUgNDBHYiBOSUMgaXMgbm90IGEgIm11c3QiLCBJIHRoaW5r IGl0IGlzIG1vcmUgYWZmb3JkYWJsZSBhcyBpdCA9CmNvc3QgbXVjaCBsZXNzIHRoZW4gYSBnb29k IGRpc2sgY29udHJvbGxlci4KCgpCdXQgbXkgY29uZnVzaW9uIGlzIHRoYXQsIGFzIHNhaWQgYnkg b3RoZXIgZmVsbG93cywgdGhlIGJlc3QgInBlcmZvcm1hbmNlIG09Cm9kZWwiIGlzIHdoZW4geW91 IHVzZSBhIGhhcmR3YXJlIFJBSURlZCBicmljayAoaS5lLjogNSBvciA2KSB0byBhc3NlbWJsZSB5 bz0KdXIgR2x1c3RlckZTLiBJbiB0aGlzIGNhc2UsIGFzIEkgd291bGQgaGF2ZSB0byBidXkgYSBn b29kIGNvbnRyb2xsZXIgYnV0IGhhPQp2ZSBsZXNzIG5ldHdvcmsgdHJhZmZpYywgdG8gbG93ZXIg dGhlIGNvc3QgSSB3b3VsZCB0aGVuIHVzZSBhIHNlcGFyYXRlIG5ldHc9Cm9yayBtYWRlIG9mIDEw R2IgTklDcyBwbHVzIHRoZSBjb250cm9sbGVyLgoKCk1vYWNpcgoKCgo+Cj4gPiBMZSA4IGFvP3Qg MjAxNyA/IDA0OjA4LCBGRVJOQU5ETyBGUkVESUFOSSA8ZmVybmFuZG8uZnJlZGlhbmlAdXB4LmNv bT4gYQo+ID9jcml0IDoKPgo+ID4gRXZlbiBpZiB5b3UgaGF2ZSBhIEhhcmR3YXJlIFJBSUQgQ29u dHJvbGxlciB3aXRoIFdyaXRlYmFjayBjYWNoZSB5b3UKPiB3aWxsIGhhdmUgYSBzaWduaWZpY2Fu dCBwZXJmb3JtYW5jZSBwZW5hbHR5IGFuZCBtYXkgbm90IGZ1bGx5IHVzZSBhbGwgdGhlCj4gcmVz b3VyY2VzIHlvdSBtZW50aW9uZWQgeW91IGhhdmUuCj4gPgo+Cj4gTm9wZSBhZ2Fpbixmcm9tIG15 IGV4cGVyaWVuY2Ugd2l0aCBIUCBTbWFydCBBcnJheSBhbmQgd3JpdGUgYmFjayBjYWNoZSwKPiB3 cml0ZSwgdGhhdCBnb2VzIGluIHRoZSBjYWNoZSwgYXJlIGV2ZW4gZmFzdGVyIHRoYXQgcmVhZCB0 aGF0IG11c3QgZ29lcyB0PQpvCj4gdGhlIGRpc2tzLiBvZiBjb3Vyc2UgaWYgdGhlIHdyaXRlIGFy ZSB0b28gZmFzdCBhbmQgdG8gYmlnLCB0aGV5IHdpbGwgb3Zlcgo+IG92ZXJmbG93IHRoZSBjYWNo ZS4gQnV0IG9uIHRvZGF5cyBjb250cm9sbGVyIHRoZXkgYXJlIG11bHRpLWdpZ2FieXRlIGNhY2g9 CmUsCj4geW91IG11c3Qgd3JpdGUgYSBsb3QgdG8gZmlsbCB0aGVtLiBBbmQgaWYgeW91IGNhbiBh ZmZvcmQgNDBHYiBjYXJkLCB5b3UgYz0KYW4KPiBhZmZvcmQgZGVjZW50IGNvbnRyb2xsZXIuCj4K ClRoZSBsYXN0IHNlbnRlbmNlIHJhaXNlcyBhbiBleGNlbGxlbnQgcG9pbnQ6IGJhbGFuY2UgeW91 ciByZXNvdXJjZXMuIERvbid0CnNwZW5kIGEgZm9ydHVuZSBvbiBvbmUgY29tcG9uZW50IHdoaWxl IGFub3RoZXIgd2lsbCBlbmQgdXAgYmVpbmcgeW91cgpib3R0bGVuZWNrLgpTdG9yYWdlIGlzIHVz dWFsbHkgdGhlIHNsb3dlc3QgbGluayBpbiB0aGUgY2hhaW4uIEkgcGVyc29uYWxseSBiZWxpZXZl IHRoYXQKc3BlbmRpbmcgdGhlIG1vbmV5IG9uIE5WTWUgZHJpdmVzIG1ha2VzIG1vcmUgc2Vuc2Ug dGhhbiA0MEdiIChleGNlcHQgWzFdLAp3aGljaCBpcyBzdXNwaWNpb3VzbHkgY2hlYXAhKQoKWS4K WzFdIGh0dHA6Ly9hLmNvLzRoc0NUcUcKCgotLV8wMDBfREI2UDE5ME1CMDI4MEFEODg4MDMyQUI0 QTYyMzJENTc2Qzg4QTBEQjZQMTkwTUIwMjgwRVVSUF8KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7 IGNoYXJzZXQ9Imlzby04ODU5LTEiCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1w cmludGFibGUKCjxodG1sPgo8aGVhZD4KPG1ldGEgaHR0cC1lcXVpdj0zRCJDb250ZW50LVR5cGUi IGNvbnRlbnQ9M0QidGV4dC9odG1sOyBjaGFyc2V0PTNEaXNvLTg4NTktPQoxIj4KPHN0eWxlIHR5 cGU9M0QidGV4dC9jc3MiIHN0eWxlPTNEImRpc3BsYXk6bm9uZTsiPjwhLS0gUCB7bWFyZ2luLXRv cDowO21hcmdpPQpuLWJvdHRvbTowO30gLS0+PC9zdHlsZT4KPC9oZWFkPgo8Ym9keSBkaXI9M0Qi bHRyIj4KPGRpdiBpZD0zRCJkaXZ0YWdkZWZhdWx0d3JhcHBlciIgc3R5bGU9M0QiZm9udC1zaXpl OjEycHQ7Y29sb3I6IzAwMDAwMDtmb250PQotZmFtaWx5OkNhbGlicmksSGVsdmV0aWNhLHNhbnMt c2VyaWY7IiBkaXI9M0QibHRyIj4KPHA+T2ssIHRoZSA0MEdiIE5JQyB0aGF0IEkgZ290IHdlcmUg Zm9yIGZyZWUuIEJ1dCBhbnl3YXksIGlmIHlvdSB3ZXJlIHdvcmtpPQpuZyB3aXRoIDYgSEREICYj NDM7IDEgU1NEIHBlciBzZXJ2ZXIsIHRoZW4geW91IGdldCAyMSBkaXNrcyBvbiB5b3VyIGNsdXN0 ZXI9Ci4gQXMgZGF0YSBpbiBhIEpCT0Qgd2lsbCBiZSBidWlsdCBhbGwgb3ZlciB0aGUgbmV0d29y aywgdGhlbiBpdCBjYW4gYmUgcmVhbD0KbHkgaW50ZW5zaXZlIGVzcGVjaWFsbHkgZGVwZW5kaW5n IG9uCiB0aGUgbnVtYmVyIG9mIHJlcGxpY2FzIHlvdSBjaG9vc2UgZm9yIHlvdXIgbmVlZHMuIEFs c28sIHdoZW4gbW92aW5nIGEgVk0gYT0KbGl2ZSB5b3UgbXVzdCB0cmFuc2ZlciB0aGUgbWVtb3J5 IGNvbnRlbnRzIG9mIGEgVk0gdG8gYW5vdGhlciBub2RlIChqdXN0IHRoPQppbmsgYWJvdXQgbW92 aW5nIGEgVk0gd2l0aCAzMkdCIFJBTSkuIEFsbCB0b2dldGhlciwgaXQgY2FuIGJlIGEgcXVpdGUg bGFyZ2U9CiBjaHVuayBvZiBkYXRhIG1vdmluZyBvdmVyIHRoZSBuZXR3b3JrCiBhbGwgdGhlIHRp bWUuIFdoaWxlIDQwR2IgTklDIGlzIG5vdCBhICZxdW90O211c3QmcXVvdDssIEkgdGhpbmsgaXQg aXMgbW9yZT0KIGFmZm9yZGFibGUgYXMgaXQgY29zdCBtdWNoIGxlc3MgdGhlbiBhIGdvb2QgZGlz ayBjb250cm9sbGVyLjwvcD4KPHA+PGJyPgo8L3A+CjxwPkJ1dCBteSBjb25mdXNpb24gaXMgdGhh dCwgYXMgc2FpZCBieSBvdGhlciBmZWxsb3dzLCB0aGUgYmVzdCAmcXVvdDtwZXJmbz0Kcm1hbmNl IG1vZGVsJnF1b3Q7IGlzIHdoZW4geW91IHVzZSBhIGhhcmR3YXJlIFJBSURlZCBicmljayAoaS5l LjogNSBvciA2KSB0PQpvIGFzc2VtYmxlIHlvdXIgR2x1c3RlckZTLiBJbiB0aGlzIGNhc2UsIGFz IEkgd291bGQgaGF2ZSB0byBidXkgYSBnb29kIGNvbnQ9CnJvbGxlciBidXQgaGF2ZSBsZXNzIG5l dHdvcmsgdHJhZmZpYywgdG8gbG93ZXIKIHRoZSBjb3N0IEkgd291bGQgdGhlbiB1c2UgYSBzZXBh cmF0ZSBuZXR3b3JrIG1hZGUgb2YgMTBHYiBOSUNzIHBsdXMgdGhlIGNvPQpudHJvbGxlci48L3A+ CjxwPjxicj4KPC9wPgo8cD5Nb2FjaXI8YnI+CjwvcD4KPGRpdiBzdHlsZT0zRCJjb2xvcjogcmdi KDQ5LCA1NSwgNTcpOyI+PGZvbnQgc2l6ZT0zRCIyIj48c3BhbiBzdHlsZT0zRCJmb250PQotc2l6 ZToxMHB0OyI+CjxkaXYgY2xhc3M9M0QiUGxhaW5UZXh0Ij48YnI+Cjxicj4KPGJyPgomZ3Q7PGJy PgomZ3Q7ICZndDsgTGUgOCBhbz90IDIwMTcgPyAwNDowOCwgRkVSTkFORE8gRlJFRElBTkkgJmx0 O2Zlcm5hbmRvLmZyZWRpYW5pQHU9CnB4LmNvbSZndDsgYTxicj4KJmd0OyA/Y3JpdCA6PGJyPgom Z3Q7PGJyPgomZ3Q7ICZndDsgRXZlbiBpZiB5b3UgaGF2ZSBhIEhhcmR3YXJlIFJBSUQgQ29udHJv bGxlciB3aXRoIFdyaXRlYmFjayBjYWNoZSA9CnlvdTxicj4KJmd0OyB3aWxsIGhhdmUgYSBzaWdu aWZpY2FudCBwZXJmb3JtYW5jZSBwZW5hbHR5IGFuZCBtYXkgbm90IGZ1bGx5IHVzZSBhbGwgPQp0 aGU8YnI+CiZndDsgcmVzb3VyY2VzIHlvdSBtZW50aW9uZWQgeW91IGhhdmUuPGJyPgomZ3Q7ICZn dDs8YnI+CiZndDs8YnI+CiZndDsgTm9wZSBhZ2Fpbixmcm9tIG15IGV4cGVyaWVuY2Ugd2l0aCBI UCBTbWFydCBBcnJheSBhbmQgd3JpdGUgYmFjayBjYWNoZT0KLDxicj4KJmd0OyB3cml0ZSwgdGhh dCBnb2VzIGluIHRoZSBjYWNoZSwgYXJlIGV2ZW4gZmFzdGVyIHRoYXQgcmVhZCB0aGF0IG11c3Qg Z29lPQpzIHRvPGJyPgomZ3Q7IHRoZSBkaXNrcy4gb2YgY291cnNlIGlmIHRoZSB3cml0ZSBhcmUg dG9vIGZhc3QgYW5kIHRvIGJpZywgdGhleSB3aWxsIG89CnZlcjxicj4KJmd0OyBvdmVyZmxvdyB0 aGUgY2FjaGUuIEJ1dCBvbiB0b2RheXMgY29udHJvbGxlciB0aGV5IGFyZSBtdWx0aS1naWdhYnl0 ZSBjPQphY2hlLDxicj4KJmd0OyB5b3UgbXVzdCB3cml0ZSBhIGxvdCB0byBmaWxsIHRoZW0uIEFu ZCBpZiB5b3UgY2FuIGFmZm9yZCA0MEdiIGNhcmQsIHlvPQp1IGNhbjxicj4KJmd0OyBhZmZvcmQg ZGVjZW50IGNvbnRyb2xsZXIuPGJyPgomZ3Q7PGJyPgo8YnI+ClRoZSBsYXN0IHNlbnRlbmNlIHJh aXNlcyBhbiBleGNlbGxlbnQgcG9pbnQ6IGJhbGFuY2UgeW91ciByZXNvdXJjZXMuIERvbid0PD0K YnI+CnNwZW5kIGEgZm9ydHVuZSBvbiBvbmUgY29tcG9uZW50IHdoaWxlIGFub3RoZXIgd2lsbCBl bmQgdXAgYmVpbmcgeW91cjxicj4KYm90dGxlbmVjay48YnI+ClN0b3JhZ2UgaXMgdXN1YWxseSB0 aGUgc2xvd2VzdCBsaW5rIGluIHRoZSBjaGFpbi4gSSBwZXJzb25hbGx5IGJlbGlldmUgdGhhdD0K PGJyPgpzcGVuZGluZyB0aGUgbW9uZXkgb24gTlZNZSBkcml2ZXMgbWFrZXMgbW9yZSBzZW5zZSB0 aGFuIDQwR2IgKGV4Y2VwdCBbMV0sPGI9CnI+CndoaWNoIGlzIHN1c3BpY2lvdXNseSBjaGVhcCEp PGJyPgo8YnI+ClkuPGJyPgpbMV0gPGEgaHJlZj0zRCJodHRwOi8vYS5jby80aHNDVHFHIiBpZD0z RCJMUGxuazgwMjgzNCIgcHJldmlld3JlbW92ZWQ9M0QidHI9CnVlIj5odHRwOi8vYS5jby80aHND VHFHPC9hPjxicj4KPGJyPgo8L2Rpdj4KPC9zcGFuPjwvZm9udD48L2Rpdj4KPC9kaXY+CjwvYm9k eT4KPC9odG1sPgoKLS1fMDAwX0RCNlAxOTBNQjAyODBBRDg4ODAzMkFCNEE2MjMyRDU3NkM4OEEw REI2UDE5ME1CMDI4MEVVUlBfLS0K --===============3670569142046871846==--