From justin at redfish.com.au Mon Feb 17 03:50:39 2014 Content-Type: multipart/mixed; boundary="===============4798810214453489854==" MIME-Version: 1.0 From: Justin Clacherty To: users at ovirt.org Subject: [Users] Why choose glusterfs over iscsi for oVirt storage? Date: Mon, 17 Feb 2014 08:50:34 +0000 Message-ID: --===============4798810214453489854== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --_000_C79B906B4ECB8E46A3281AC932C805502F49039Aexchangeredfish_ Content-Type: text/plain; charset=3D"us-ascii" Content-Transfer-Encoding: quoted-printable Hi, I'm just setting up some storage for use with oVirt and am wondering why I = =3D might choose glusterfs rather than just exporting a raid array as iscsi. T= =3D he oVirt team seem to be pushing gluster (though that could just be because= =3D it's a Red Hat product). Can anyone answer this one? What I have come up with is as follows. For: - easy to expand - redundancy across storage devices/easy replication - high availablility - performance - it's kind of cool :) - maintenance? Against (these are guesses): - performance? (multiple layers of filesystems everywhere - fs in = =3D vm + image on gluster + gluster + block filesystem) - complexity - maintenance? Any help here is appreciated. Also, does the underlying block level filesy= =3D stem matter here? VMs running under ovirt would be typical business applic= =3D ations - file serving (samba4), email, databases, web servers, etc. Cheers, Justin. --_000_C79B906B4ECB8E46A3281AC932C805502F49039Aexchangeredfish_ Content-Type: text/html; charset=3D"us-ascii" Content-Transfer-Encoding: quoted-printable

Hi= ,=3D

I&=3D #8217;m just setting up some storage for use with oVirt and am wondering wh= =3D y I might choose glusterfs rather than just exporting a raid array as iscsi= =3D . The oVirt team seem to be pushing gluster (though that could just b= =3D e because it’s a Red Hat product). Can anyone answer this one?<= =3D o:p>

What I have come up with is as follows.

For:

- <= =3D /span>easy to expand

- redundancy across storage devices/easy replication

- &nb= sp=3D ; high availablility

- = =3D performance

- = =3D it’s kind of cool J

- &nb= sp=3D ; maintenance?<= =3D o:p>

Against (these are guesses):

= - performance? (multiple layers of filesystems everywhere – fs in = =3D vm + image on gluster + gluster + block filesystem)

<=3D ![if !supportLists]>- &nbs= =3D p; complexity

- maintenance?

Any help here is appreciated. Also, does= t=3D he underlying block level filesystem matter here? VMs running under o= =3D virt would be typical business applications – file serving (samba4), = =3D email, databases, web servers, etc.

Cheers,

Justin.

=3D

=3D --_000_C79B906B4ECB8E46A3281AC932C805502F49039Aexchangeredfish_-- --===============4798810214453489854== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" LS1fMDAwX0M3OUI5MDZCNEVDQjhFNDZBMzI4MUFDOTMyQzgwNTUwMkY0OTAzOUFleGNoYW5nZXJl ZGZpc2hfCkNvbnRlbnQtVHlwZTogdGV4dC9wbGFpbjsgY2hhcnNldD0idXMtYXNjaWkiCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCkhpLAoKSSdtIGp1c3Qgc2V0 dGluZyB1cCBzb21lIHN0b3JhZ2UgZm9yIHVzZSB3aXRoIG9WaXJ0IGFuZCBhbSB3b25kZXJpbmcg d2h5IEkgPQptaWdodCBjaG9vc2UgZ2x1c3RlcmZzIHJhdGhlciB0aGFuIGp1c3QgZXhwb3J0aW5n IGEgcmFpZCBhcnJheSBhcyBpc2NzaS4gIFQ9CmhlIG9WaXJ0IHRlYW0gc2VlbSB0byBiZSBwdXNo aW5nIGdsdXN0ZXIgKHRob3VnaCB0aGF0IGNvdWxkIGp1c3QgYmUgYmVjYXVzZT0KIGl0J3MgYSBS ZWQgSGF0IHByb2R1Y3QpLiAgQ2FuIGFueW9uZSBhbnN3ZXIgdGhpcyBvbmU/CgpXaGF0IEkgaGF2 ZSBjb21lIHVwIHdpdGggaXMgYXMgZm9sbG93cy4KCkZvcjoKCi0gICAgICAgICAgZWFzeSB0byBl eHBhbmQKCi0gICAgICAgICAgcmVkdW5kYW5jeSBhY3Jvc3Mgc3RvcmFnZSBkZXZpY2VzL2Vhc3kg cmVwbGljYXRpb24KCi0gICAgICAgICAgaGlnaCBhdmFpbGFibGlsaXR5CgotICAgICAgICAgIHBl cmZvcm1hbmNlCgotICAgICAgICAgIGl0J3Mga2luZCBvZiBjb29sIDopCgotICAgICAgICAgIG1h aW50ZW5hbmNlPwoKQWdhaW5zdCAodGhlc2UgYXJlIGd1ZXNzZXMpOgoKLSAgICAgICAgICBwZXJm b3JtYW5jZT8gKG11bHRpcGxlIGxheWVycyBvZiBmaWxlc3lzdGVtcyBldmVyeXdoZXJlIC0gZnMg aW4gPQp2bSArIGltYWdlIG9uIGdsdXN0ZXIgKyBnbHVzdGVyICsgYmxvY2sgZmlsZXN5c3RlbSkK Ci0gICAgICAgICAgY29tcGxleGl0eQoKLSAgICAgICAgICBtYWludGVuYW5jZT8KCkFueSBoZWxw IGhlcmUgaXMgYXBwcmVjaWF0ZWQuICBBbHNvLCBkb2VzIHRoZSB1bmRlcmx5aW5nIGJsb2NrIGxl dmVsIGZpbGVzeT0Kc3RlbSBtYXR0ZXIgaGVyZT8gIFZNcyBydW5uaW5nIHVuZGVyIG92aXJ0IHdv dWxkIGJlIHR5cGljYWwgYnVzaW5lc3MgYXBwbGljPQphdGlvbnMgLSBmaWxlIHNlcnZpbmcgKHNh bWJhNCksIGVtYWlsLCBkYXRhYmFzZXMsIHdlYiBzZXJ2ZXJzLCBldGMuCgpDaGVlcnMsCkp1c3Rp bi4KCgoKCi0tXzAwMF9DNzlCOTA2QjRFQ0I4RTQ2QTMyODFBQzkzMkM4MDU1MDJGNDkwMzlBZXhj aGFuZ2VyZWRmaXNoXwpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsgY2hhcnNldD0idXMtYXNjaWki CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCjxodG1sIHhtbG5z OnY9M0QidXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89M0QidXJuOnNjaGVt YXMtbWljcj0Kb3NvZnQtY29tOm9mZmljZTpvZmZpY2UiIHhtbG5zOnc9M0QidXJuOnNjaGVtYXMt bWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgPQp4bWxuczptPTNEImh0dHA6Ly9zY2hlbWFzLm1p Y3Jvc29mdC5jb20vb2ZmaWNlLzIwMDQvMTIvb21tbCIgeG1sbnM9M0QiaHR0cDo9Ci8vd3d3Lncz Lm9yZy9UUi9SRUMtaHRtbDQwIj48aGVhZD48bWV0YSBodHRwLWVxdWl2PTNEQ29udGVudC1UeXBl IGNvbnRlbnQ9Cj0zRCJ0ZXh0L2h0bWw7IGNoYXJzZXQ9M0R1cy1hc2NpaSI+PG1ldGEgbmFtZT0z REdlbmVyYXRvciBjb250ZW50PTNEIk1pY3Jvcz0Kb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1 bSkiPjxzdHlsZT48IS0tCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8KQGZvbnQtZmFjZQoJe2ZvbnQt ZmFtaWx5OldpbmdkaW5nczsKCXBhbm9zZS0xOjUgMCAwIDAgMCAwIDAgMCAwIDA7fQpAZm9udC1m YWNlCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0aCI7CglwYW5vc2UtMToyIDQgNSAzIDUgNCA2 IDMgMiA0O30KQGZvbnQtZmFjZQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7CglwYW5vc2UtMToyIDE1 IDUgMiAyIDIgNCAzIDIgNDt9Ci8qIFN0eWxlIERlZmluaXRpb25zICovCnAuTXNvTm9ybWFsLCBs aS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwKCXttYXJnaW46MGNtOwoJbWFyZ2luLWJvdHRvbTou MDAwMXB0OwoJZm9udC1zaXplOjExLjBwdDsKCWZvbnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1z ZXJpZiI7Cgltc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUzt9CmE6bGluaywgc3Bhbi5Nc29IeXBl cmxpbmsKCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7Cgljb2xvcjojMDU2M0MxOwoJdGV4dC1kZWNv cmF0aW9uOnVuZGVybGluZTt9CmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZAoJ e21zby1zdHlsZS1wcmlvcml0eTo5OTsKCWNvbG9yOiM5NTRGNzI7Cgl0ZXh0LWRlY29yYXRpb246 dW5kZXJsaW5lO30KcC5Nc29MaXN0UGFyYWdyYXBoLCBsaS5Nc29MaXN0UGFyYWdyYXBoLCBkaXYu TXNvTGlzdFBhcmFncmFwaAoJe21zby1zdHlsZS1wcmlvcml0eTozNDsKCW1hcmdpbi10b3A6MGNt OwoJbWFyZ2luLXJpZ2h0OjBjbTsKCW1hcmdpbi1ib3R0b206MGNtOwoJbWFyZ2luLWxlZnQ6MzYu MHB0OwoJbWFyZ2luLWJvdHRvbTouMDAwMXB0OwoJZm9udC1zaXplOjExLjBwdDsKCWZvbnQtZmFt aWx5OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7Cgltc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUzt9 CnNwYW4uRW1haWxTdHlsZTE3Cgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtY29tcG9zZTsKCWZv bnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7Cgljb2xvcjp3aW5kb3d0ZXh0O30KLk1z b0NocERlZmF1bHQKCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsKCWZvbnQtZmFtaWx5OiJD YWxpYnJpIiwic2Fucy1zZXJpZiI7Cgltc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUzt9CkBwYWdl IFdvcmRTZWN0aW9uMQoJe3NpemU6NjEyLjBwdCA3OTIuMHB0OwoJbWFyZ2luOjcyLjBwdCA3Mi4w cHQgNzIuMHB0IDcyLjBwdDt9CmRpdi5Xb3JkU2VjdGlvbjEKCXtwYWdlOldvcmRTZWN0aW9uMTt9 Ci8qIExpc3QgRGVmaW5pdGlvbnMgKi8KQGxpc3QgbDAKCXttc28tbGlzdC1pZDo5MTM0Njc4MzM7 Cgltc28tbGlzdC10eXBlOmh5YnJpZDsKCW1zby1saXN0LXRlbXBsYXRlLWlkczotMTkzODQxMDU4 OCA5NDcwNTA3ODYgMjAxOTE2NDE5IDIwMTkxNjQyMSAyMDE5MTY0MTcgPQoyMDE5MTY0MTkgMjAx OTE2NDIxIDIwMTkxNjQxNyAyMDE5MTY0MTkgMjAxOTE2NDIxO30KQGxpc3QgbDA6bGV2ZWwxCgl7 bXNvLWxldmVsLXN0YXJ0LWF0OjA7Cgltc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDpidWxsZXQ7Cglt c28tbGV2ZWwtdGV4dDotOwoJbXNvLWxldmVsLXRhYi1zdG9wOm5vbmU7Cgltc28tbGV2ZWwtbnVt YmVyLXBvc2l0aW9uOmxlZnQ7Cgl0ZXh0LWluZGVudDotMTguMHB0OwoJZm9udC1mYW1pbHk6IkNh bGlicmkiLCJzYW5zLXNlcmlmIjsKCW1zby1mYXJlYXN0LWZvbnQtZmFtaWx5OkNhbGlicmk7Cglt c28tYmlkaS1mb250LWZhbWlseToiVGltZXMgTmV3IFJvbWFuIjt9CkBsaXN0IGwwOmxldmVsMgoJ e21zby1sZXZlbC1udW1iZXItZm9ybWF0OmJ1bGxldDsKCW1zby1sZXZlbC10ZXh0Om87Cgltc28t bGV2ZWwtdGFiLXN0b3A6bm9uZTsKCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsKCXRl eHQtaW5kZW50Oi0xOC4wcHQ7Cglmb250LWZhbWlseToiQ291cmllciBOZXciO30KQGxpc3QgbDA6 bGV2ZWwzCgl7bXNvLWxldmVsLW51bWJlci1mb3JtYXQ6YnVsbGV0OwoJbXNvLWxldmVsLXRleHQ6 XEYwQTc7Cgltc28tbGV2ZWwtdGFiLXN0b3A6bm9uZTsKCW1zby1sZXZlbC1udW1iZXItcG9zaXRp b246bGVmdDsKCXRleHQtaW5kZW50Oi0xOC4wcHQ7Cglmb250LWZhbWlseTpXaW5nZGluZ3M7fQpA bGlzdCBsMDpsZXZlbDQKCXttc28tbGV2ZWwtbnVtYmVyLWZvcm1hdDpidWxsZXQ7Cgltc28tbGV2 ZWwtdGV4dDpcRjBCNzsKCW1zby1sZXZlbC10YWItc3RvcDpub25lOwoJbXNvLWxldmVsLW51bWJl ci1wb3NpdGlvbjpsZWZ0OwoJdGV4dC1pbmRlbnQ6LTE4LjBwdDsKCWZvbnQtZmFtaWx5OlN5bWJv bDt9CkBsaXN0IGwwOmxldmVsNQoJe21zby1sZXZlbC1udW1iZXItZm9ybWF0OmJ1bGxldDsKCW1z by1sZXZlbC10ZXh0Om87Cgltc28tbGV2ZWwtdGFiLXN0b3A6bm9uZTsKCW1zby1sZXZlbC1udW1i ZXItcG9zaXRpb246bGVmdDsKCXRleHQtaW5kZW50Oi0xOC4wcHQ7Cglmb250LWZhbWlseToiQ291 cmllciBOZXciO30KQGxpc3QgbDA6bGV2ZWw2Cgl7bXNvLWxldmVsLW51bWJlci1mb3JtYXQ6YnVs bGV0OwoJbXNvLWxldmVsLXRleHQ6XEYwQTc7Cgltc28tbGV2ZWwtdGFiLXN0b3A6bm9uZTsKCW1z by1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsKCXRleHQtaW5kZW50Oi0xOC4wcHQ7Cglmb250 LWZhbWlseTpXaW5nZGluZ3M7fQpAbGlzdCBsMDpsZXZlbDcKCXttc28tbGV2ZWwtbnVtYmVyLWZv cm1hdDpidWxsZXQ7Cgltc28tbGV2ZWwtdGV4dDpcRjBCNzsKCW1zby1sZXZlbC10YWItc3RvcDpu b25lOwoJbXNvLWxldmVsLW51bWJlci1wb3NpdGlvbjpsZWZ0OwoJdGV4dC1pbmRlbnQ6LTE4LjBw dDsKCWZvbnQtZmFtaWx5OlN5bWJvbDt9CkBsaXN0IGwwOmxldmVsOAoJe21zby1sZXZlbC1udW1i ZXItZm9ybWF0OmJ1bGxldDsKCW1zby1sZXZlbC10ZXh0Om87Cgltc28tbGV2ZWwtdGFiLXN0b3A6 bm9uZTsKCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsKCXRleHQtaW5kZW50Oi0xOC4w cHQ7Cglmb250LWZhbWlseToiQ291cmllciBOZXciO30KQGxpc3QgbDA6bGV2ZWw5Cgl7bXNvLWxl dmVsLW51bWJlci1mb3JtYXQ6YnVsbGV0OwoJbXNvLWxldmVsLXRleHQ6XEYwQTc7Cgltc28tbGV2 ZWwtdGFiLXN0b3A6bm9uZTsKCW1zby1sZXZlbC1udW1iZXItcG9zaXRpb246bGVmdDsKCXRleHQt aW5kZW50Oi0xOC4wcHQ7Cglmb250LWZhbWlseTpXaW5nZGluZ3M7fQpvbAoJe21hcmdpbi1ib3R0 b206MGNtO30KdWwKCXttYXJnaW4tYm90dG9tOjBjbTt9Ci0tPjwvc3R5bGU+PCEtLVtpZiBndGUg bXNvIDldPjx4bWw+CjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9M0QiZWRpdCIgc3BpZG1heD0zRCIx MDI2IiAvPgo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4KPG86c2hh cGVsYXlvdXQgdjpleHQ9M0QiZWRpdCI+CjxvOmlkbWFwIHY6ZXh0PTNEImVkaXQiIGRhdGE9M0Qi MSIgLz4KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0tPjwvaGVhZD48Ym9keSBsYW5n PTNERU4tQVUgbGluaz0zRCIjMDU2M0MxPQoiIHZsaW5rPTNEIiM5NTRGNzIiPjxkaXYgY2xhc3M9 M0RXb3JkU2VjdGlvbjE+PHAgY2xhc3M9M0RNc29Ob3JtYWw+SGksPG86cD49CjwvbzpwPjwvcD48 cCBjbGFzcz0zRE1zb05vcm1hbD48bzpwPiZuYnNwOzwvbzpwPjwvcD48cCBjbGFzcz0zRE1zb05v cm1hbD5JJj0KIzgyMTc7bSBqdXN0IHNldHRpbmcgdXAgc29tZSBzdG9yYWdlIGZvciB1c2Ugd2l0 aCBvVmlydCBhbmQgYW0gd29uZGVyaW5nIHdoPQp5IEkgbWlnaHQgY2hvb3NlIGdsdXN0ZXJmcyBy YXRoZXIgdGhhbiBqdXN0IGV4cG9ydGluZyBhIHJhaWQgYXJyYXkgYXMgaXNjc2k9Ci4mbmJzcDsg VGhlIG9WaXJ0IHRlYW0gc2VlbSB0byBiZSBwdXNoaW5nIGdsdXN0ZXIgKHRob3VnaCB0aGF0IGNv dWxkIGp1c3QgYj0KZSBiZWNhdXNlIGl0JiM4MjE3O3MgYSBSZWQgSGF0IHByb2R1Y3QpLiZuYnNw OyBDYW4gYW55b25lIGFuc3dlciB0aGlzIG9uZT88PQpvOnA+PC9vOnA+PC9wPjxwIGNsYXNzPTNE TXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPjxwIGNsYXNzPTNETXNvTm9ybWE9Cmw+V2hh dCBJIGhhdmUgY29tZSB1cCB3aXRoIGlzIGFzIGZvbGxvd3MuPG86cD48L286cD48L3A+PHAgY2xh c3M9M0RNc29Ob3JtYT0KbD48bzpwPiZuYnNwOzwvbzpwPjwvcD48cCBjbGFzcz0zRE1zb05vcm1h bD5Gb3I6PG86cD48L286cD48L3A+PHAgY2xhc3M9M0RNPQpzb0xpc3RQYXJhZ3JhcGggc3R5bGU9 M0QndGV4dC1pbmRlbnQ6LTE4LjBwdDttc28tbGlzdDpsMCBsZXZlbDEgbGZvMSc+PCFbaWY9CiAh c3VwcG9ydExpc3RzXT48c3BhbiBzdHlsZT0zRCdtc28tbGlzdDpJZ25vcmUnPi08c3BhbiBzdHls ZT0zRCdmb250OjcuMHB0ID0KIlRpbWVzIE5ldyBSb21hbiInPiZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyA8PQovc3Bhbj48L3NwYW4+PCFbZW5k aWZdPmVhc3kgdG8gZXhwYW5kPG86cD48L286cD48L3A+PHAgY2xhc3M9M0RNc29MaXN0UGFyYWc9 CnJhcGggc3R5bGU9M0QndGV4dC1pbmRlbnQ6LTE4LjBwdDttc28tbGlzdDpsMCBsZXZlbDEgbGZv MSc+PCFbaWYgIXN1cHBvcnRMaT0Kc3RzXT48c3BhbiBzdHlsZT0zRCdtc28tbGlzdDpJZ25vcmUn Pi08c3BhbiBzdHlsZT0zRCdmb250OjcuMHB0ICJUaW1lcyBOZXcgPQpSb21hbiInPiZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyA8L3NwYW4+PC9z cGE9Cm4+PCFbZW5kaWZdPnJlZHVuZGFuY3kgYWNyb3NzIHN0b3JhZ2UgZGV2aWNlcy9lYXN5IHJl cGxpY2F0aW9uPG86cD48L286cD48Lz0KcD48cCBjbGFzcz0zRE1zb0xpc3RQYXJhZ3JhcGggc3R5 bGU9M0QndGV4dC1pbmRlbnQ6LTE4LjBwdDttc28tbGlzdDpsMCBsZXZlPQpsMSBsZm8xJz48IVtp ZiAhc3VwcG9ydExpc3RzXT48c3BhbiBzdHlsZT0zRCdtc28tbGlzdDpJZ25vcmUnPi08c3BhbiBz dHlsZT0KPTNEJ2ZvbnQ6Ny4wcHQgIlRpbWVzIE5ldyBSb21hbiInPiZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwPQo7Jm5ic3A7Jm5ic3A7IDwvc3Bhbj48L3NwYW4+PCFb ZW5kaWZdPmhpZ2ggYXZhaWxhYmxpbGl0eTxvOnA+PC9vOnA+PC9wPjxwIGM9Cmxhc3M9M0RNc29M aXN0UGFyYWdyYXBoIHN0eWxlPTNEJ3RleHQtaW5kZW50Oi0xOC4wcHQ7bXNvLWxpc3Q6bDAgbGV2 ZWwxIGxmbz0KMSc+PCFbaWYgIXN1cHBvcnRMaXN0c10+PHNwYW4gc3R5bGU9M0QnbXNvLWxpc3Q6 SWdub3JlJz4tPHNwYW4gc3R5bGU9M0QnZm9uPQp0OjcuMHB0ICJUaW1lcyBOZXcgUm9tYW4iJz4m bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDs9CiZuYnNwOyA8 L3NwYW4+PC9zcGFuPjwhW2VuZGlmXT5wZXJmb3JtYW5jZTxvOnA+PC9vOnA+PC9wPjxwIGNsYXNz PTNETXNvTGlzdD0KUGFyYWdyYXBoIHN0eWxlPTNEJ3RleHQtaW5kZW50Oi0xOC4wcHQ7bXNvLWxp c3Q6bDAgbGV2ZWwxIGxmbzEnPjwhW2lmICFzdXBwPQpvcnRMaXN0c10+PHNwYW4gc3R5bGU9M0Qn bXNvLWxpc3Q6SWdub3JlJz4tPHNwYW4gc3R5bGU9M0QnZm9udDo3LjBwdCAiVGltZXM9CiBOZXcg Um9tYW4iJz4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsgPC9zcGFuPj0KPC9zcGFuPjwhW2VuZGlmXT5pdCYjODIxNztzIGtpbmQgb2YgY29vbCA8 c3BhbiBzdHlsZT0zRCdmb250LWZhbWlseTpXaW5nZGluPQpncyc+Sjwvc3Bhbj48bzpwPjwvbzpw PjwvcD48cCBjbGFzcz0zRE1zb0xpc3RQYXJhZ3JhcGggc3R5bGU9M0QndGV4dC1pbmRlbnQ9Cjot MTguMHB0O21zby1saXN0OmwwIGxldmVsMSBsZm8xJz48IVtpZiAhc3VwcG9ydExpc3RzXT48c3Bh biBzdHlsZT0zRCdtc28tbD0KaXN0Oklnbm9yZSc+LTxzcGFuIHN0eWxlPTNEJ2ZvbnQ6Ny4wcHQg IlRpbWVzIE5ldyBSb21hbiInPiZuYnNwOyZuYnNwOyZuYnNwPQo7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7IDwvc3Bhbj48L3NwYW4+PCFbZW5kaWZdPm1haW50ZW5hbmNlPzw9 Cm86cD48L286cD48L3A+PHAgY2xhc3M9M0RNc29Ob3JtYWw+PG86cD4mbmJzcDs8L286cD48L3A+ PHAgY2xhc3M9M0RNc29Ob3JtYT0KbD5BZ2FpbnN0ICh0aGVzZSBhcmUgZ3Vlc3Nlcyk6PG86cD48 L286cD48L3A+PHAgY2xhc3M9M0RNc29MaXN0UGFyYWdyYXBoIHN0PQp5bGU9M0QndGV4dC1pbmRl bnQ6LTE4LjBwdDttc28tbGlzdDpsMCBsZXZlbDEgbGZvMSc+PCFbaWYgIXN1cHBvcnRMaXN0c10+ PHM9CnBhbiBzdHlsZT0zRCdtc28tbGlzdDpJZ25vcmUnPi08c3BhbiBzdHlsZT0zRCdmb250Ojcu MHB0ICJUaW1lcyBOZXcgUm9tYW4iJz0KPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyA8L3NwYW4+PC9zcGFuPjwhW2VuPQpkaWZdPnBlcmZvcm1h bmNlPyAobXVsdGlwbGUgbGF5ZXJzIG9mIGZpbGVzeXN0ZW1zIGV2ZXJ5d2hlcmUgJiM4MjExOyBm cyBpbiA9CnZtICsgaW1hZ2Ugb24gZ2x1c3RlciArIGdsdXN0ZXIgKyBibG9jayBmaWxlc3lzdGVt KTxvOnA+PC9vOnA+PC9wPjxwIGNsYXNzPQo9M0RNc29MaXN0UGFyYWdyYXBoIHN0eWxlPTNEJ3Rl eHQtaW5kZW50Oi0xOC4wcHQ7bXNvLWxpc3Q6bDAgbGV2ZWwxIGxmbzEnPjw9CiFbaWYgIXN1cHBv cnRMaXN0c10+PHNwYW4gc3R5bGU9M0QnbXNvLWxpc3Q6SWdub3JlJz4tPHNwYW4gc3R5bGU9M0Qn Zm9udDo3Lj0KMHB0ICJUaW1lcyBOZXcgUm9tYW4iJz4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzPQpwOyA8L3NwYW4+PC9zcGFuPjwhW2VuZGlmXT5j b21wbGV4aXR5PG86cD48L286cD48L3A+PHAgY2xhc3M9M0RNc29MaXN0UGFyYWc9CnJhcGggc3R5 bGU9M0QndGV4dC1pbmRlbnQ6LTE4LjBwdDttc28tbGlzdDpsMCBsZXZlbDEgbGZvMSc+PCFbaWYg IXN1cHBvcnRMaT0Kc3RzXT48c3BhbiBzdHlsZT0zRCdtc28tbGlzdDpJZ25vcmUnPi08c3BhbiBz dHlsZT0zRCdmb250OjcuMHB0ICJUaW1lcyBOZXcgPQpSb21hbiInPiZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyA8L3NwYW4+PC9zcGE9Cm4+PCFb ZW5kaWZdPm1haW50ZW5hbmNlPzxvOnA+PC9vOnA+PC9wPjxwIGNsYXNzPTNETXNvTm9ybWFsPjxv OnA+Jm5ic3A7PC9vOj0KcD48L3A+PHAgY2xhc3M9M0RNc29Ob3JtYWw+QW55IGhlbHAgaGVyZSBp cyBhcHByZWNpYXRlZC4mbmJzcDsgQWxzbywgZG9lcyB0PQpoZSB1bmRlcmx5aW5nIGJsb2NrIGxl dmVsIGZpbGVzeXN0ZW0gbWF0dGVyIGhlcmU/Jm5ic3A7IFZNcyBydW5uaW5nIHVuZGVyIG89CnZp cnQgd291bGQgYmUgdHlwaWNhbCBidXNpbmVzcyBhcHBsaWNhdGlvbnMgJiM4MjExOyBmaWxlIHNl cnZpbmcgKHNhbWJhNCksID0KZW1haWwsIGRhdGFiYXNlcywgd2ViIHNlcnZlcnMsIGV0Yy48bzpw PjwvbzpwPjwvcD48cCBjbGFzcz0zRE1zb05vcm1hbD48bzpwPQo+Jm5ic3A7PC9vOnA+PC9wPjxw IGNsYXNzPTNETXNvTm9ybWFsPkNoZWVycyw8bzpwPjwvbzpwPjwvcD48cCBjbGFzcz0zRE1zb049 Cm9ybWFsPkp1c3Rpbi48bzpwPjwvbzpwPjwvcD48cCBjbGFzcz0zRE1zb05vcm1hbD48bzpwPiZu YnNwOzwvbzpwPjwvcD48cCBjbD0KYXNzPTNETXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PC9w PjxwIGNsYXNzPTNETXNvTm9ybWFsPjxvOnA+Jm5ic3A7PC9vOnA+PQo8L3A+PC9kaXY+PC9ib2R5 PjwvaHRtbD49CgotLV8wMDBfQzc5QjkwNkI0RUNCOEU0NkEzMjgxQUM5MzJDODA1NTAyRjQ5MDM5 QWV4Y2hhbmdlcmVkZmlzaF8tLQo= --===============4798810214453489854==-- From sdainard at miovision.com Thu Feb 20 14:27:21 2014 Content-Type: multipart/mixed; boundary="===============5056716293462507661==" MIME-Version: 1.0 From: Steve Dainard To: users at ovirt.org Subject: Re: [Users] Why choose glusterfs over iscsi for oVirt storage? Date: Thu, 20 Feb 2014 14:27:20 -0500 Message-ID: In-Reply-To: C79B906B4ECB8E46A3281AC932C805502F49039A@exchange.redfish.local --===============5056716293462507661== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Mon, Feb 17, 2014 at 3:50 AM, Justin Clacherty = wrote: > Hi, > > > > I'm just setting up some storage for use with oVirt and am wondering why I > might choose glusterfs rather than just exporting a raid array as iscsi. > The oVirt team seem to be pushing gluster (though that could just be > because it's a Red Hat product). Can anyone answer this one? > If you're contemplating software iscsi then you probably aren't working on a production setup that requires high availability or data consistency. If you are, you should take a deep dive on what your single point of failures are. If you're actually talking about a SAN lun, then I would imagine you aren't the storage admin, and can leave the particulars in their hands. Its funny, in RH's current supported version of RHEV, native gluster storage isn't even an option. This is a pretty new project, and if you follow the bread crumbs on distributed file systems like glusterfs or ceph you'll start to see the future advantages. You can also figure that seeing as both projects are spearheaded by RH dev's that there is likely a lot of cross-talk and feature requests making both projects better, and who wouldn't promote a better solution? > > > What I have come up with is as follows. > > > > For: > There is too much to cover in any of these advantages, you're going to need to do a lot of research on each of these 'features' if you want to use them successfully. > - easy to expand > > - redundancy across storage devices/easy replication > > - high availablility > > - performance > > - it's kind of cool J > > - maintenance? > > > > Against (these are guesses): > > - performance? (multiple layers of filesystems everywhere - fs > in vm + image on gluster + gluster + block filesystem) > Its worse than just a ton of software layers. NAS protocols seem to be focused on data consistency, which means they don't write async to storage. iscsi is typically async and has much better throughput, but also a greater chance for data loss or corruption. Technically you can achieve the same level of performance as iscsi using NFS (backed by glusterfs if you like) but you need to set options on the volume to allow async writes. > - complexity > If you're doing storage properly, the underpinnings are always complex unless you're paying someone else to do it (read: SAN / managed service provider). Research multipath and HA on software iscsi and you'll see what I mean. > - maintenance? > > > > Any help here is appreciated. Also, does the underlying block level > filesystem matter here? VMs running under ovirt would be typical business > applications - file serving (samba4), email, databases, web servers, etc. > There is a lot to answer here and I don't have all the answers. Take a look at the gluster docs for underlying file system requirements. Any block device should work. Specifically I'll mention that the glusterfs team doesn't suggest hosting db's on glusterfs - many small reads/writes are not one of glusters strong points. > > > Cheers, > > Justin. > > > > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============5056716293462507661== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj4KPGJyPjxicj48ZGl2 IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gTW9uLCBGZWIgMTcsIDIwMTQgYXQgMzo1MCBBTSwgSnVz dGluIENsYWNoZXJ0eSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpqdXN0aW5A cmVkZmlzaC5jb20uYXUiIHRhcmdldD0iX2JsYW5rIiBvbmNsaWNrPSJ3aW5kb3cub3BlbigmIzM5 O2h0dHBzOi8vbWFpbC5nb29nbGUuY29tL21haWwvP3ZpZXc9Y20mYW1wO3RmPTEmYW1wO3RvPWp1 c3RpbkByZWRmaXNoLmNvbS5hdSZhbXA7Y2M9JmFtcDtiY2M9JmFtcDtzdT0mYW1wO2JvZHk9JiMz OTssJiMzOTtfYmxhbmsmIzM5Oyk7cmV0dXJuIGZhbHNlOyI+anVzdGluQHJlZGZpc2guY29tLmF1 PC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj4KPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3Rl IiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7cGFk ZGluZy1sZWZ0OjFleCI+PGRpdiBsYW5nPSJFTi1BVSIgbGluaz0iIzA1NjNDMSIgdmxpbms9IiM5 NTRGNzIiPjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+SGksPHU+PC91Pjx1PjwvdT48L3A+PHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PiZuYnNwOzx1PjwvdT48L3A+CjxwIGNsYXNzPSJNc29O b3JtYWwiPkkmcnNxdW87bSBqdXN0IHNldHRpbmcgdXAgc29tZSBzdG9yYWdlIGZvciB1c2Ugd2l0 aCBvVmlydCBhbmQgYW0gd29uZGVyaW5nIHdoeSBJIG1pZ2h0IGNob29zZSBnbHVzdGVyZnMgcmF0 aGVyIHRoYW4ganVzdCBleHBvcnRpbmcgYSByYWlkIGFycmF5IGFzIGlzY3NpLiZuYnNwOyBUaGUg b1ZpcnQgdGVhbSBzZWVtIHRvIGJlIHB1c2hpbmcgZ2x1c3RlciAodGhvdWdoIHRoYXQgY291bGQg anVzdCBiZSBiZWNhdXNlIGl0JnJzcXVvO3MgYSBSZWQgSGF0IHByb2R1Y3QpLiZuYnNwOyBDYW4g YW55b25lIGFuc3dlciB0aGlzIG9uZT88L3A+CjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2 Pjxicj48L2Rpdj48ZGl2PklmIHlvdSYjMzk7cmUgY29udGVtcGxhdGluZyBzb2Z0d2FyZSBpc2Nz aSB0aGVuIHlvdSBwcm9iYWJseSBhcmVuJiMzOTt0IHdvcmtpbmcgb24gYSBwcm9kdWN0aW9uIHNl dHVwIHRoYXQgcmVxdWlyZXMgaGlnaCBhdmFpbGFiaWxpdHkgb3IgZGF0YSBjb25zaXN0ZW5jeS4g SWYgeW91IGFyZSwgeW91IHNob3VsZCB0YWtlIGEgZGVlcCBkaXZlIG9uIHdoYXQgeW91ciBzaW5n bGUgcG9pbnQgb2YgZmFpbHVyZXMgYXJlLiBJZiB5b3UmIzM5O3JlIGFjdHVhbGx5IHRhbGtpbmcg YWJvdXQgYSBTQU4gbHVuLCB0aGVuIEkgd291bGQgaW1hZ2luZSB5b3UgYXJlbiYjMzk7dCB0aGUg c3RvcmFnZSBhZG1pbiwgYW5kIGNhbiBsZWF2ZSB0aGUgcGFydGljdWxhcnMgaW4gdGhlaXIgaGFu ZHMuPC9kaXY+CjxkaXY+PGJyPjwvZGl2PjxkaXY+SXRzIGZ1bm55LCBpbiBSSCYjMzk7cyBjdXJy ZW50IHN1cHBvcnRlZCB2ZXJzaW9uIG9mIFJIRVYsIG5hdGl2ZSBnbHVzdGVyIHN0b3JhZ2UgaXNu JiMzOTt0IGV2ZW4gYW4gb3B0aW9uLiBUaGlzIGlzIGEgcHJldHR5IG5ldyBwcm9qZWN0LCBhbmQg aWYgeW91IGZvbGxvdyB0aGUgYnJlYWQgY3J1bWJzIG9uIGRpc3RyaWJ1dGVkIGZpbGUgc3lzdGVt cyBsaWtlIGdsdXN0ZXJmcyBvciBjZXBoIHlvdSYjMzk7bGwgc3RhcnQgdG8gc2VlIHRoZSBmdXR1 cmUgYWR2YW50YWdlcy4gWW91IGNhbiBhbHNvIGZpZ3VyZSB0aGF0IHNlZWluZyBhcyBib3RoIHBy b2plY3RzIGFyZSBzcGVhcmhlYWRlZCBieSBSSCBkZXYmIzM5O3MgdGhhdCB0aGVyZSBpcyBsaWtl bHkgYSBsb3Qgb2YgY3Jvc3MtdGFsayBhbmQgZmVhdHVyZSByZXF1ZXN0cyBtYWtpbmcgYm90aCBw cm9qZWN0cyBiZXR0ZXIsIGFuZCB3aG8gd291bGRuJiMzOTt0IHByb21vdGUgYSBiZXR0ZXIgc29s dXRpb24/PC9kaXY+CjxkaXY+Jm5ic3A7PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1 b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4ICNjY2Mgc29saWQ7 cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBsYW5nPSJFTi1BVSIgbGluaz0iIzA1NjNDMSIgdmxpbms9 IiM5NTRGNzIiPjxkaXY+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91Pjx1PjwvdT48L3A+PHAg Y2xhc3M9Ik1zb05vcm1hbCI+Cjx1PjwvdT4mbmJzcDs8dT48L3U+PC9wPjxwIGNsYXNzPSJNc29O b3JtYWwiPldoYXQgSSBoYXZlIGNvbWUgdXAgd2l0aCBpcyBhcyBmb2xsb3dzLjx1PjwvdT48dT48 L3U+PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT4mbmJzcDs8dT48L3U+PC9wPjxwIGNs YXNzPSJNc29Ob3JtYWwiPkZvcjo8L3A+PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJy PjwvZGl2PjxkaXY+VGhlcmUgaXMgdG9vIG11Y2ggdG8gY292ZXIgaW4gYW55IG9mIHRoZXNlIGFk dmFudGFnZXMsIHlvdSYjMzk7cmUgZ29pbmcgdG8gbmVlZCB0byBkbyBhIGxvdCBvZiByZXNlYXJj aCBvbiBlYWNoIG9mIHRoZXNlICYjMzk7ZmVhdHVyZXMmIzM5OyBpZiB5b3Ugd2FudCB0byB1c2Ug dGhlbSBzdWNjZXNzZnVsbHkuJm5ic3A7PC9kaXY+CjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9x dW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlk O3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgbGFuZz0iRU4tQVUiIGxpbms9IiMwNTYzQzEiIHZsaW5r PSIjOTU0RjcyIj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT48dT48L3U+PC9wPjxw Pjx1PjwvdT48c3Bhbj4tPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMgTmV3IFJv bWFuJnF1b3Q7Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsgPC9zcGFuPjwvc3Bhbj48dT48L3U+ZWFzeSB0byBleHBhbmQ8dT48L3U+PHU+PC91 PjwvcD4KPHA+PHU+PC91PjxzcGFuPi08c3BhbiBzdHlsZT0iZm9udDo3LjBwdCAmcXVvdDtUaW1l cyBOZXcgUm9tYW4mcXVvdDsiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyA8L3NwYW4+PC9zcGFuPjx1PjwvdT5yZWR1bmRhbmN5IGFjcm9zcyBz dG9yYWdlIGRldmljZXMvZWFzeSByZXBsaWNhdGlvbjx1PjwvdT48dT48L3U+PC9wPjxwPjx1Pjwv dT48c3Bhbj4tPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsgPC9zcGFuPjwvc3Bhbj48dT48L3U+aGlnaCBhdmFpbGFibGlsaXR5PHU+PC91Pjx1PjwvdT48 L3A+CjxwPjx1PjwvdT48c3Bhbj4tPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMg TmV3IFJvbWFuJnF1b3Q7Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsgPC9zcGFuPjwvc3Bhbj48dT48L3U+cGVyZm9ybWFuY2U8dT48L3U+PHU+ PC91PjwvcD48cD48dT48L3U+PHNwYW4+LTxzcGFuIHN0eWxlPSJmb250OjcuMHB0ICZxdW90O1Rp bWVzIE5ldyBSb21hbiZxdW90OyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7IDwvc3Bhbj48L3NwYW4+PHU+PC91Pml0JnJzcXVvO3Mga2luZCBv ZiBjb29sIDxzcGFuIHN0eWxlPSJmb250LWZhbWlseTpXaW5nZGluZ3MiPko8L3NwYW4+PHU+PC91 Pjx1PjwvdT48L3A+CjxwPjx1PjwvdT48c3Bhbj4tPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1 b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgPC9zcGFuPjwvc3Bhbj48dT48L3U+bWFpbnRlbmFuY2U/ PHU+PC91Pjx1PjwvdT48L3A+PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PiZuYnNwOzx1Pjwv dT48L3A+PHAgY2xhc3M9Ik1zb05vcm1hbCI+QWdhaW5zdCAodGhlc2UgYXJlIGd1ZXNzZXMpOjx1 PjwvdT48dT48L3U+PC9wPgo8cD48dT48L3U+PHNwYW4+LTxzcGFuIHN0eWxlPSJmb250OjcuMHB0 ICZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IDwvc3Bhbj48L3NwYW4+PHU+PC91PnBlcmZvcm1h bmNlPyAobXVsdGlwbGUgbGF5ZXJzIG9mIGZpbGVzeXN0ZW1zIGV2ZXJ5d2hlcmUgJm5kYXNoOyBm cyBpbiB2bSArIGltYWdlIG9uIGdsdXN0ZXIgKyBnbHVzdGVyICsgYmxvY2sgZmlsZXN5c3RlbSk8 L3A+PC9kaXY+CjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pkl0cyB3b3JzZSB0aGFuIGp1c3QgYSB0 b24gb2Ygc29mdHdhcmUgbGF5ZXJzLiBOQVMgcHJvdG9jb2xzIHNlZW0gdG8gYmUgZm9jdXNlZCBv biBkYXRhIGNvbnNpc3RlbmN5LCB3aGljaCBtZWFucyB0aGV5IGRvbiYjMzk7dCB3cml0ZSBhc3lu YyB0byBzdG9yYWdlLiBpc2NzaSBpcyB0eXBpY2FsbHkgYXN5bmMgYW5kIGhhcyBtdWNoIGJldHRl ciB0aHJvdWdocHV0LCBidXQgYWxzbyBhIGdyZWF0ZXIgY2hhbmNlIGZvciBkYXRhIGxvc3Mgb3Ig Y29ycnVwdGlvbi4gVGVjaG5pY2FsbHkgeW91IGNhbiBhY2hpZXZlIHRoZSBzYW1lIGxldmVsIG9m IHBlcmZvcm1hbmNlIGFzIGlzY3NpIHVzaW5nIE5GUyAoYmFja2VkIGJ5IGdsdXN0ZXJmcyBpZiB5 b3UgbGlrZSkgYnV0IHlvdSBuZWVkIHRvIHNldCBvcHRpb25zIG9uIHRoZSB2b2x1bWUgdG8gYWxs b3cgYXN5bmMgd3JpdGVzLjwvZGl2Pgo8ZGl2PiZuYnNwOzwvZGl2PjxibG9ja3F1b3RlIGNsYXNz PSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAj Y2NjIHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgbGFuZz0iRU4tQVUiIGxpbms9IiMwNTYz QzEiIHZsaW5rPSIjOTU0RjcyIj48ZGl2PjxwPjx1PjwvdT48dT48L3U+PC9wPjxwPjx1PjwvdT48 c3Bhbj4tPHNwYW4gc3R5bGU9ImZvbnQ6Ny4wcHQgJnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7 Ij4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsg PC9zcGFuPjwvc3Bhbj48dT48L3U+Y29tcGxleGl0eTwvcD4KPC9kaXY+PC9kaXY+PC9ibG9ja3F1 b3RlPjxkaXY+SWYgeW91JiMzOTtyZSBkb2luZyBzdG9yYWdlIHByb3Blcmx5LCB0aGUgdW5kZXJw aW5uaW5ncyBhcmUgYWx3YXlzIGNvbXBsZXggdW5sZXNzIHlvdSYjMzk7cmUgcGF5aW5nIHNvbWVv bmUgZWxzZSB0byBkbyBpdCAocmVhZDogU0FOIC8gbWFuYWdlZCBzZXJ2aWNlIHByb3ZpZGVyKS4g UmVzZWFyY2ggbXVsdGlwYXRoIGFuZCBIQSBvbiBzb2Z0d2FyZSBpc2NzaSBhbmQgeW91JiMzOTts bCBzZWUgd2hhdCBJIG1lYW4uPC9kaXY+CjxkaXY+Jm5ic3A7PC9kaXY+PGJsb2NrcXVvdGUgY2xh c3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjAgMCAwIC44ZXg7Ym9yZGVyLWxlZnQ6MXB4 ICNjY2Mgc29saWQ7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBsYW5nPSJFTi1BVSIgbGluaz0iIzA1 NjNDMSIgdmxpbms9IiM5NTRGNzIiPjxkaXY+PHA+PHU+PC91Pjx1PjwvdT48L3A+PHA+PHU+PC91 PjxzcGFuPi08c3BhbiBzdHlsZT0iZm9udDo3LjBwdCAmcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVv dDsiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyA8L3NwYW4+PC9zcGFuPjx1PjwvdT5tYWludGVuYW5jZT88dT48L3U+PHU+PC91PjwvcD4KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHU+PC91PiZuYnNwOzx1PjwvdT48L3A+PHAgY2xhc3M9Ik1zb05v cm1hbCI+QW55IGhlbHAgaGVyZSBpcyBhcHByZWNpYXRlZC4mbmJzcDsgQWxzbywgZG9lcyB0aGUg dW5kZXJseWluZyBibG9jayBsZXZlbCBmaWxlc3lzdGVtIG1hdHRlciBoZXJlPyZuYnNwOyBWTXMg cnVubmluZyB1bmRlciBvdmlydCB3b3VsZCBiZSB0eXBpY2FsIGJ1c2luZXNzIGFwcGxpY2F0aW9u cyAmbmRhc2g7IGZpbGUgc2VydmluZyAoc2FtYmE0KSwgZW1haWwsIGRhdGFiYXNlcywgd2ViIHNl cnZlcnMsIGV0Yy48L3A+CjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48L2Rpdj48 ZGl2PlRoZXJlIGlzIGEgbG90IHRvIGFuc3dlciBoZXJlIGFuZCBJIGRvbiYjMzk7dCBoYXZlIGFs bCB0aGUgYW5zd2Vycy4gVGFrZSBhIGxvb2sgYXQgdGhlIGdsdXN0ZXIgZG9jcyBmb3IgdW5kZXJs eWluZyBmaWxlIHN5c3RlbSByZXF1aXJlbWVudHMuIEFueSBibG9jayBkZXZpY2Ugc2hvdWxkIHdv cmsuIFNwZWNpZmljYWxseSBJJiMzOTtsbCBtZW50aW9uIHRoYXQgdGhlIGdsdXN0ZXJmcyB0ZWFt IGRvZXNuJiMzOTt0IHN1Z2dlc3QgaG9zdGluZyBkYiYjMzk7cyBvbiBnbHVzdGVyZnMgLSBtYW55 IHNtYWxsIHJlYWRzL3dyaXRlcyBhcmUgbm90IG9uZSBvZiBnbHVzdGVycyBzdHJvbmcgcG9pbnRz LjwvZGl2Pgo8ZGl2PiZuYnNwOzwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIg c3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRp bmctbGVmdDoxZXgiPjxkaXYgbGFuZz0iRU4tQVUiIGxpbms9IiMwNTYzQzEiIHZsaW5rPSIjOTU0 RjcyIj48ZGl2PjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT48dT48L3U+PC9wPjxwIGNsYXNz PSJNc29Ob3JtYWwiPgo8dT48L3U+Jm5ic3A7PHU+PC91PjwvcD48cCBjbGFzcz0iTXNvTm9ybWFs Ij5DaGVlcnMsPHU+PC91Pjx1PjwvdT48L3A+PHAgY2xhc3M9Ik1zb05vcm1hbCI+SnVzdGluLjx1 PjwvdT48dT48L3U+PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT4mbmJzcDs8dT48L3U+ PC9wPjxwIGNsYXNzPSJNc29Ob3JtYWwiPjx1PjwvdT4mbmJzcDs8dT48L3U+PC9wPjxwIGNsYXNz PSJNc29Ob3JtYWwiPjx1PjwvdT4mbmJzcDs8dT48L3U+PC9wPgo8L2Rpdj48L2Rpdj48YnI+X19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1h aWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgb25jbGljaz0i d2luZG93Lm9wZW4oJiMzOTtodHRwczovL21haWwuZ29vZ2xlLmNvbS9tYWlsLz92aWV3PWNtJmFt cDt0Zj0xJmFtcDt0bz1Vc2Vyc0BvdmlydC5vcmcmYW1wO2NjPSZhbXA7YmNjPSZhbXA7c3U9JmFt cDtib2R5PSYjMzk7LCYjMzk7X2JsYW5rJiMzOTspO3JldHVybiBmYWxzZTsiPlVzZXJzQG92aXJ0 Lm9yZzwvYT48YnI+Cgo8YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz dGluZm8vdXNlcnMiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxt YW4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+PC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rp dj48L2Rpdj4K --===============5056716293462507661==--