Hello, I'd like to ask community, what is the best way to use oVirt in the following hardware configuration: Three servers connected 1GB network. Each server - 32 threads, 256GB RAM, 4TB RAID. Please note that a local storage and an 1GB network is a typical hardware configuration for almost any dedicated hosting. Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes. Does somebody have such setup? Thanks
Hi, We have all nodes like you (5 currently), and have them setup as NFS instead of local storage. You have slight overhead because it's NFS, but not so much, and it allows you to migrate VMs and disks, even if it's a manual disk, eg no HA will be possible. So yeah, to me, it's a good option given your setup. Best,
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Pavel Gashev Sent: jeudi 3 mars 2016 21:44 To: users@ovirt.org Subject: [ovirt-users] Multi-node cluster with local storage
Hello,
I'd like to ask community, what is the best way to use oVirt in the following hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB RAM, 4TB RAID.
Please note that a local storage and an 1GB network is a typical hardware configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes.
Does somebody have such setup?
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_000_2e54f1f30c004db3ab5621ff68a2ffeeacroniscom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Christophe, Where is the engine running? Did you use one of NFSs for hosted-engine? Thank you From: Christophe TREFOIS <christophe.trefois@uni.lu> Sent: Mar 4, 2016 2:09 AM To: Pavel Gashev; users@ovirt.org Subject: RE: Multi-node cluster with local storage Hi, We have all nodes like you (5 currently), and have them setup as NFS instea= d of local storage. You have slight overhead because it's NFS, but not so much, and it allows y= ou to migrate VMs and disks, even if it's a manual disk, eg no HA will be p= ossible. So yeah, to me, it's a good option given your setup. Best,
-----Original Message----- From: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] On Behalf Of Pavel Gashev Sent: jeudi 3 mars 2016 21:44 To: users@ovirt.org Subject: [ovirt-users] Multi-node cluster with local storage
Hello,
I'd like to ask community, what is the best way to use oVirt in the follo= wing hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB RAM, 4TB RAID.
Please note that a local storage and an 1GB network is a typical hardware configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that th= e only way to use oVirt in a three-node cluster is to share local storages = over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes.
Does somebody have such setup?
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--_000_2e54f1f30c004db3ab5621ff68a2ffeeacroniscom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft Exchange Server"> <!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; pad= ding-left: 4pt; border-left: #800000 2px solid; } --></style> </head> <body> <style type=3D"text/css"> <!-- --> </style> <div> <div style=3D"font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.= 0pt; color:#1F497D"> <p dir=3D"ltr">Christophe,</p> <p dir=3D"ltr">Where is the engine running? Did you use one of NFSs for hos= ted-engine?</p> <p dir=3D"ltr">Thank you</p> <div id=3D"x_signature-x" style=3D"font-family:Calibri,Arial,Helvetica,sans= -serif; font-size:12.0pt; color:#1F497D"> <br> </div> </div> <div id=3D"x_quoted_header" style=3D"clear:both"><br> <div style=3D"border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0c= m 0cm 0cm"> <span style=3D"font-size:11.0pt; font-family:'Calibri','sans-serif'"><b>Fro= m:</b> Christophe TREFOIS <christophe.trefois@uni.lu><br> <b>Sent:</b> Mar 4, 2016 2:09 AM<br> <b>To:</b> Pavel Gashev; users@ovirt.org<br> <b>Subject:</b> RE: Multi-node cluster with local storage<br> </span></div> </div> <br type=3D"attribution"> </div> <font size=3D"2"><span style=3D"font-size:10pt;"> <div class=3D"PlainText">Hi,<br> <br> We have all nodes like you (5 currently), and have them setup as NFS instea= d of local storage.<br> <br> You have slight overhead because it's NFS, but not so much, and it allows y= ou to migrate VMs and disks, even if it's a manual disk, eg no HA will be p= ossible.<br> <br> So yeah, to me, it's a good option given your setup.<br> <br> Best,<br> <br> > -----Original Message-----<br> > From: users-bounces@ovirt.org [<a href=3D"mailto:users-bounces@ovirt.o= rg">mailto:users-bounces@ovirt.org</a>] On Behalf<br> > Of Pavel Gashev<br> > Sent: jeudi 3 mars 2016 21:44<br> > To: users@ovirt.org<br> > Subject: [ovirt-users] Multi-node cluster with local storage<br> > <br> > Hello,<br> > <br> > I'd like to ask community, what is the best way to use oVirt in the fo= llowing<br> > hardware configuration:<br> > <br> > Three servers connected 1GB network. Each server - 32 threads, 256GB R= AM,<br> > 4TB RAID.<br> > <br> > Please note that a local storage and an 1GB network is a typical hardw= are<br> > configuration for almost any dedicated hosting.<br> > <br> > Unfortunately, oVirt doesn't support multi-node local storage clusters= .<br> > And Gluster/CEPH doesn't work well over 1G network. It looks like that= the<br> > only way to use oVirt in a three-node cluster is to share local storag= es over<br> > NFS. At least it makes possible to migrate VMs and move disks among<br=
> hardware nodes.<br> > <br> > Does somebody have such setup?<br> > <br> > Thanks<br> > _______________________________________________<br> > Users mailing list<br> > Users@ovirt.org<br> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/users">http://lists= .ovirt.org/mailman/listinfo/users</a><br> </div> </span></font> </body> </html> --_000_2e54f1f30c004db3ab5621ff68a2ffeeacroniscom_--
--_000_A1DB10E79F804D5A865F352A4C7ECE43unilu_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgUGF2ZWwsDQoNCldlIHNldHVwIGFuIGV4dHJhIE5GUyBvbiBvbmUgb2YgdGhlIGhvc3RzLCBh bmQgc2V0dXAgYXMgaG9zdGVkLWVuZ2luZS4NCg0KVGhhdCB3YXksIGl0IHdvcmtzIGZvciB1cy4N Cg0KVXNpbmcgR2x1c3RlciAzLXdheSB3aXRoIENUREIgd2FzIG91ciBpbml0aWFsIHBsYW4sIGJ1 dCB3ZSBtb3ZlZCBhd2F5IGZyb20gaXQgYXMgaXQgc2VlbXMgdW5uZWNhc3NpcnkgY29tcGxpY2F0 ZWQuDQoNCktpbmQgcmVhZ3JkcywNCg0K4oCUDQpDaHJpc3RvcGhlDQoNCkRyIENocmlzdG9waGUg VHJlZm9pcywgRGlwbC4tSW5nLg0KVGVjaG5pY2FsIFNwZWNpYWxpc3QgLyBQb3N0LURvYw0KDQpV TklWRVJTSVTDiSBEVSBMVVhFTUJPVVJHDQoNCkxVWEVNQk9VUkcgQ0VOVFJFIEZPUiBTWVNURU1T IEJJT01FRElDSU5FDQpDYW1wdXMgQmVsdmFsIHwgSG91c2Ugb2YgQmlvbWVkaWNpbmUNCjYsIGF2 ZW51ZSBkdSBTd2luZw0KTC00MzY3IEJlbHZhdXgNClQ6ICszNTIgNDYgNjYgNDQgNjEyNA0KRjog KzM1MiA0NiA2NiA0NCA2OTQ5DQpodHRwOi8vd3d3LnVuaS5sdS9sY3NiDQoNCltGYWNlYm9va108 aHR0cHM6Ly93d3cuZmFjZWJvb2suY29tL3RyZWZleD4gIFtUd2l0dGVyXSA8aHR0cHM6Ly90d2l0 dGVyLmNvbS9UcmVmZXg+ICAgW0dvb2dsZSBQbHVzXSA8aHR0cHM6Ly9wbHVzLmdvb2dsZS5jb20v K0NocmlzdG9waGVUcmVmb2lzLz4gICBbTGlua2VkaW5dIDxodHRwczovL3d3dy5saW5rZWRpbi5j b20vaW4vdHJlZm9pc2NocmlzdG9waGU+ICAgW3NreXBlXSA8aHR0cDovL3NreXBlOlRyZWZleD9j YWxsPg0KDQotLS0tDQpUaGlzIG1lc3NhZ2UgaXMgY29uZmlkZW50aWFsIGFuZCBtYXkgY29udGFp biBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLg0KSXQgaXMgaW50ZW5kZWQgZm9yIHRoZSBuYW1lZCBy ZWNpcGllbnQgb25seS4NCklmIHlvdSByZWNlaXZlIGl0IGluIGVycm9yIHBsZWFzZSBub3RpZnkg bWUgYW5kIHBlcm1hbmVudGx5IGRlbGV0ZSB0aGUgb3JpZ2luYWwgbWVzc2FnZSBhbmQgYW55IGNv cGllcy4NCi0tLS0NCg0KDQoNCk9uIDA0IE1hciAyMDE2LCBhdCAwNzo0NiwgUGF2ZWwgR2FzaGV2 IDxQYXhAYWNyb25pcy5jb208bWFpbHRvOlBheEBhY3JvbmlzLmNvbT4+IHdyb3RlOg0KDQoNCkNo cmlzdG9waGUsDQoNCldoZXJlIGlzIHRoZSBlbmdpbmUgcnVubmluZz8gRGlkIHlvdSB1c2Ugb25l IG9mIE5GU3MgZm9yIGhvc3RlZC1lbmdpbmU/DQoNClRoYW5rIHlvdQ0KDQoNCkZyb206IENocmlz dG9waGUgVFJFRk9JUyA8Y2hyaXN0b3BoZS50cmVmb2lzQHVuaS5sdTxtYWlsdG86Y2hyaXN0b3Bo ZS50cmVmb2lzQHVuaS5sdT4+DQpTZW50OiBNYXIgNCwgMjAxNiAyOjA5IEFNDQpUbzogUGF2ZWwg R2FzaGV2OyB1c2Vyc0BvdmlydC5vcmc8bWFpbHRvOnVzZXJzQG92aXJ0Lm9yZz4NClN1YmplY3Q6 IFJFOiBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBzdG9yYWdlDQoNCkhpLA0KDQpXZSBo YXZlIGFsbCBub2RlcyBsaWtlIHlvdSAoNSBjdXJyZW50bHkpLCBhbmQgaGF2ZSB0aGVtIHNldHVw IGFzIE5GUyBpbnN0ZWFkIG9mIGxvY2FsIHN0b3JhZ2UuDQoNCllvdSBoYXZlIHNsaWdodCBvdmVy aGVhZCBiZWNhdXNlIGl0J3MgTkZTLCBidXQgbm90IHNvIG11Y2gsIGFuZCBpdCBhbGxvd3MgeW91 IHRvIG1pZ3JhdGUgVk1zIGFuZCBkaXNrcywgZXZlbiBpZiBpdCdzIGEgbWFudWFsIGRpc2ssIGVn IG5vIEhBIHdpbGwgYmUgcG9zc2libGUuDQoNClNvIHllYWgsIHRvIG1lLCBpdCdzIGEgZ29vZCBv cHRpb24gZ2l2ZW4geW91ciBzZXR1cC4NCg0KQmVzdCwNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3Nh Z2UtLS0tLQ0KPiBGcm9tOiB1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnMtYm91 bmNlc0BvdmlydC5vcmc+IFttYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmddIE9uIEJlaGFs Zg0KPiBPZiBQYXZlbCBHYXNoZXYNCj4gU2VudDogamV1ZGkgMyBtYXJzIDIwMTYgMjE6NDQNCj4g VG86IHVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnNAb3ZpcnQub3JnPg0KPiBTdWJqZWN0OiBb b3ZpcnQtdXNlcnNdIE11bHRpLW5vZGUgY2x1c3RlciB3aXRoIGxvY2FsIHN0b3JhZ2UNCj4NCj4g SGVsbG8sDQo+DQo+IEknZCBsaWtlIHRvIGFzayBjb21tdW5pdHksIHdoYXQgaXMgdGhlIGJlc3Qg d2F5IHRvIHVzZSBvVmlydCBpbiB0aGUgZm9sbG93aW5nDQo+IGhhcmR3YXJlIGNvbmZpZ3VyYXRp b246DQo+DQo+IFRocmVlIHNlcnZlcnMgY29ubmVjdGVkIDFHQiBuZXR3b3JrLiBFYWNoIHNlcnZl ciAtIDMyIHRocmVhZHMsIDI1NkdCIFJBTSwNCj4gNFRCIFJBSUQuDQo+DQo+IFBsZWFzZSBub3Rl IHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFuZCBhbiAxR0IgbmV0d29yayBpcyBhIHR5cGljYWwgaGFy ZHdhcmUNCj4gY29uZmlndXJhdGlvbiBmb3IgYWxtb3N0IGFueSBkZWRpY2F0ZWQgaG9zdGluZy4N Cj4NCj4gVW5mb3J0dW5hdGVseSwgb1ZpcnQgZG9lc24ndCBzdXBwb3J0IG11bHRpLW5vZGUgbG9j YWwgc3RvcmFnZSBjbHVzdGVycy4NCj4gQW5kIEdsdXN0ZXIvQ0VQSCBkb2Vzbid0IHdvcmsgd2Vs bCBvdmVyIDFHIG5ldHdvcmsuIEl0IGxvb2tzIGxpa2UgdGhhdCB0aGUNCj4gb25seSB3YXkgdG8g dXNlIG9WaXJ0IGluIGEgdGhyZWUtbm9kZSBjbHVzdGVyIGlzIHRvIHNoYXJlIGxvY2FsIHN0b3Jh Z2VzIG92ZXINCj4gTkZTLiBBdCBsZWFzdCBpdCBtYWtlcyBwb3NzaWJsZSB0byBtaWdyYXRlIFZN cyBhbmQgbW92ZSBkaXNrcyBhbW9uZw0KPiBoYXJkd2FyZSBub2Rlcy4NCj4NCj4gRG9lcyBzb21l Ym9keSBoYXZlIHN1Y2ggc2V0dXA/DQo+DQo+IFRoYW5rcw0KPiBfX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBVc2VycyBtYWlsaW5nIGxpc3QNCj4gVXNl cnNAb3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+DQo+IGh0dHA6Ly9saXN0cy5vdmly dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycw0KDQo= --_000_A1DB10E79F804D5A865F352A4C7ECE43unilu_ Content-Type: text/html; charset="utf-8" Content-ID: <C647DAAEEBEFBF41BBCB33ED7B678FCD@uni.lux> Content-Transfer-Encoding: base64 PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5IHN0eWxlPSJ3b3JkLXdy YXA6IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJy ZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPg0KSGkgUGF2ZWwsDQo8ZGl2IGNsYXNz PSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5XZSBzZXR1cCBhbiBleHRy YSBORlMgb24gb25lIG9mIHRoZSBob3N0cywgYW5kIHNldHVwIGFzIGhvc3RlZC1lbmdpbmUuPC9k aXY+DQo8ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5U aGF0IHdheSwgaXQgd29ya3MgZm9yIHVzLjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9 IiI+DQo8L2Rpdj4NCjxkaXYgY2xhc3M9IiI+VXNpbmcgR2x1c3RlciAzLXdheSB3aXRoIENUREIg d2FzIG91ciBpbml0aWFsIHBsYW4sIGJ1dCB3ZSBtb3ZlZCBhd2F5IGZyb20gaXQgYXMgaXQgc2Vl bXMgdW5uZWNhc3NpcnkgY29tcGxpY2F0ZWQuPC9kaXY+DQo8ZGl2IGNsYXNzPSIiPjxiciBjbGFz cz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5LaW5kIHJlYWdyZHMsPC9kaXY+DQo8ZGl2IGNs YXNzPSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj7igJQ8L2Rpdj4NCjxk aXYgY2xhc3M9IiI+Q2hyaXN0b3BoZTxiciBjbGFzcz0iIj4NCjxkaXYgY2xhc3M9IiI+DQo8ZGl2 IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBo YW5zOiBhdXRvOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFu c2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdpZG93czogYXV0bzsgd29yZC1zcGFj aW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgd29yZC13cmFwOiBicmVh ay13b3JkOyAtd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IC13ZWJraXQtbGluZS1icmVhazogYWZ0 ZXItd2hpdGUtc3BhY2U7IiBjbGFzcz0iIj4NCjxwIHN0eWxlPSJmb250LWZhbWlseTogQXJpYWws IHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTBwdDsgbGluZS1oZWlnaHQ6IDE2cHg7IGNvbG9yOiBy Z2IoMzMsIDMzLCAzMyk7IiBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJmb250LXdlaWdodDogYm9s ZDsgY29sb3I6IHJnYig2MSwgNTksIDU5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+RHIg Q2hyaXN0b3BoZSBUcmVmb2lzLCBEaXBsLi1JbmcuPC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1j b252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVk LXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNz PSIiPjwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6IHJnYig2MSwgNTks IDU5KTsgZGlzcGxheTogaW5saW5lOyBmb250LXNpemU6IDcuNXB0OyIgY2xhc3M9IiI+VGVjaG5p Y2FsIFNwZWNpYWxpc3QgLyBQb3N0LURvYzwvc3Bhbj48L3A+DQo8cCBzdHlsZT0iZm9udC1mYW1p bHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDcuNXB0OyBsaW5lLWhlaWdodDogMTZw eDsiIGNsYXNzPSIiPg0KPHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0OiBib2xkOyBjb2xvcjogcmdi KDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5VTklWRVJTSVTDiSBEVSBM VVhFTUJPVVJHPC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJkaXNwbGF5OiBpbmxp bmU7IiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+DQo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtd2Vp Z2h0OiBib2xkOyBjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFz cz0iIj5MVVhFTUJPVVJHIENFTlRSRSBGT1IgU1lTVEVNUyBCSU9NRURJQ0lORTwvc3Bhbj48YnIg Y2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6IHJnYig2MSwgNTksIDU5KTsgZGlzcGxheTog aW5saW5lOyIgY2xhc3M9IiI+Q2FtcHVzIEJlbHZhbCB8IEhvdXNlIG9mIEJpb21lZGljaW5lPHNw YW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNz PSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3Bh biBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj42LCBhdmVudWUgZHUgU3dpbmcmbmJzcDs8 L3NwYW4+PGJyIGNsYXNzPSIiPg0KTC00MzY3IEJlbHZhdXg8c3BhbiBjbGFzcz0iQXBwbGUtY29u dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z cGFjZSI+Jm5ic3A7PC9zcGFuPjwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29s b3I6IHJnYig2MSwgNTksIDU5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+VDo8c3BhbiBj bGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxzcGFuIHN0 eWxlPSJjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj4m IzQzOzM1MiA0NiA2NiA0NCA2MTI0PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQt c3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6IHJn Yig2MSwgNTksIDU5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+Rjo8c3BhbiBjbGFzcz0i QXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxzcGFuIHN0eWxlPSJj b2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj4mIzQzOzM1 MiA0NiA2NiA0NCA2OTQ5PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2Ui PiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8 L3NwYW4+PHNwYW4gc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxiciBjbGFzcz0i Ij4NCjwvc3Bhbj48YSBocmVmPSJodHRwOi8vd3d3LnVuaS5sdS9sY3NiIiBzdHlsZT0iY29sb3I6 IHJnYigwLCAxMDksIDE4OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPmh0dHA6Ly93d3cu dW5pLmx1L2xjc2I8L2E+PC9wPg0KPHAgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1z ZXJpZjsgZm9udC1zaXplOiAxNHB4OyBsaW5lLWhlaWdodDogMTZweDsiIGNsYXNzPSIiPg0KPGEg aHJlZj0iaHR0cHM6Ly93d3cuZmFjZWJvb2suY29tL3RyZWZleCIgc3R5bGU9ImRpc3BsYXk6IGlu bGluZTsiIGNsYXNzPSIiPjxpbWcgd2lkdGg9IjI0IiBoZWlnaHQ9IjI0IiBkYXRhLWZpbGVuYW1l PSJmYWNlYm9vay5wbmciIHNyYz0iaHR0cHM6Ly9zMy5hbWF6b25hd3MuY29tL2h0bWxzaWctYXNz ZXRzL3JvdW5kZWQvZmFjZWJvb2sucG5nIiBhbHQ9IkZhY2Vib29rIiBjbGFzcz0iIj48L2E+PHNw YW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNz PSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJodHRwczovL3R3 aXR0ZXIuY29tL1RyZWZleCIgc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxpbWcg d2lkdGg9IjI0IiBoZWlnaHQ9IjI0IiBkYXRhLWZpbGVuYW1lPSJ0d2l0dGVyLnBuZyIgc3JjPSJo dHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC90d2l0dGVyLnBu ZyIgYWx0PSJUd2l0dGVyIiBjbGFzcz0iIj48L2E+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRl ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2Ui PiZuYnNwOzwvc3Bhbj48YSBocmVmPSJodHRwczovL3BsdXMuZ29vZ2xlLmNvbS8mIzQzO0Nocmlz dG9waGVUcmVmb2lzLyIgc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxpbWcgd2lk dGg9IjI0IiBoZWlnaHQ9IjI0IiBkYXRhLWZpbGVuYW1lPSJnb29nbGVwbHVzLnBuZyIgc3JjPSJo dHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC9nb29nbGVwbHVz LnBuZyIgYWx0PSJHb29nbGUgUGx1cyIgY2xhc3M9IiI+PC9hPjxzcGFuIGNsYXNzPSJBcHBsZS1j b252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVk LXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0iaHR0cHM6Ly93d3cubGlua2VkaW4uY29tL2lu L3RyZWZvaXNjaHJpc3RvcGhlIiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGlt ZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmlsZW5hbWU9ImxpbmtlZGluLnBuZyIgc3Jj PSJodHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC9saW5rZWRp bi5wbmciIGFsdD0iTGlua2VkaW4iIGNsYXNzPSIiPjwvYT48c3BhbiBjbGFzcz0iQXBwbGUtY29u dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z cGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Imh0dHA6Ly9za3lwZTpUcmVmZXg/Y2FsbCIgc3R5 bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxpbWcgd2lkdGg9IjI0IiBoZWlnaHQ9IjI0 IiBkYXRhLWZpbGVuYW1lPSJza3lwZS5wbmciIHNyYz0iaHR0cHM6Ly9zMy5hbWF6b25hd3MuY29t L2h0bWxzaWctYXNzZXRzL3JvdW5kZWQvc2t5cGUucG5nIiBhbHQ9InNreXBlIiBjbGFzcz0iIj48 L2E+PC9wPg0KPHAgY2xhc3M9ImJhbm5lci1jb250YWluZXIiIHN0eWxlPSJmb250LWZhbWlseTog QXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsgbGluZS1oZWlnaHQ6IDE2cHg7Ij4N CjwvcD4NCjxwIHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGNvbG9yOiBy Z2IoNjEsIDU5LCA1OSk7IGZvbnQtc2l6ZTogOXB4OyBsaW5lLWhlaWdodDogMTZweDsiIGNsYXNz PSIiPg0KLS0tLTxiciBjbGFzcz0iIj4NClRoaXMgbWVzc2FnZSBpcyBjb25maWRlbnRpYWwgYW5k IG1heSBjb250YWluIHByaXZpbGVnZWQgaW5mb3JtYXRpb24uPHNwYW4gY2xhc3M9IkFwcGxlLWNv bnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCkl0IGlzIGludGVuZGVk IGZvciB0aGUgbmFtZWQgcmVjaXBpZW50IG9ubHkuPHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRl ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCklmIHlvdSByZWNlaXZlIGl0IGlu IGVycm9yIHBsZWFzZSBub3RpZnkgbWUgYW5kIHBlcm1hbmVudGx5IGRlbGV0ZSB0aGUgb3JpZ2lu YWwgbWVzc2FnZSBhbmQgYW55IGNvcGllcy48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNw YWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KLS0tLTxiciBjbGFzcz0iIj4NCjwvcD4N CiZuYnNwOzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48 L2Rpdj4NCjwvZGl2Pg0KPGJyIGNsYXNzPSIiPg0KPGRpdj4NCjxibG9ja3F1b3RlIHR5cGU9ImNp dGUiIGNsYXNzPSIiPg0KPGRpdiBjbGFzcz0iIj5PbiAwNCBNYXIgMjAxNiwgYXQgMDc6NDYsIFBh dmVsIEdhc2hldiAmbHQ7PGEgaHJlZj0ibWFpbHRvOlBheEBhY3JvbmlzLmNvbSIgY2xhc3M9IiI+ UGF4QGFjcm9uaXMuY29tPC9hPiZndDsgd3JvdGU6PC9kaXY+DQo8YnIgY2xhc3M9IkFwcGxlLWlu dGVyY2hhbmdlLW5ld2xpbmUiPg0KPGRpdiBjbGFzcz0iIj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFt aWx5OiBIZWx2ZXRpY2E7IGZvbnQtc2l6ZTogMTVweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250 LXZhcmlhbnQ6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5v cm1hbDsgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7 IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87 IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IiBjbGFz cz0iIj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBDYWxpYnJpLCBBcmlhbCwgSGVsdmV0aWNh LCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEycHQ7IGNvbG9yOiByZ2IoMzEsIDczLCAxMjUpOyIg Y2xhc3M9IiI+DQo8cCBkaXI9Imx0ciIgY2xhc3M9IiI+Q2hyaXN0b3BoZSw8L3A+DQo8cCBkaXI9 Imx0ciIgY2xhc3M9IiI+V2hlcmUgaXMgdGhlIGVuZ2luZSBydW5uaW5nPyBEaWQgeW91IHVzZSBv bmUgb2YgTkZTcyBmb3IgaG9zdGVkLWVuZ2luZT88L3A+DQo8cCBkaXI9Imx0ciIgY2xhc3M9IiI+ VGhhbmsgeW91PC9wPg0KPGRpdiBpZD0ieF9zaWduYXR1cmUteCIgc3R5bGU9ImZvbnQtZmFtaWx5 OiBDYWxpYnJpLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEycHQ7 IGNvbG9yOiByZ2IoMzEsIDczLCAxMjUpOyIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQo8L2Rp dj4NCjwvZGl2Pg0KPGRpdiBpZD0ieF9xdW90ZWRfaGVhZGVyIiBzdHlsZT0iY2xlYXI6IGJvdGg7 IiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+DQo8ZGl2IHN0eWxlPSJib3JkZXItc3R5bGU6IHNvbGlk IG5vbmUgbm9uZTsgYm9yZGVyLXRvcC1jb2xvcjogcmdiKDIyNSwgMjI1LCAyMjUpOyBib3JkZXIt dG9wLXdpZHRoOiAxcHQ7IHBhZGRpbmc6IDNwdCAwY20gMGNtOyIgY2xhc3M9IiI+DQo8c3BhbiBz dHlsZT0iZm9udC1zaXplOiAxMXB0OyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJpZjsi IGNsYXNzPSIiPjxiIGNsYXNzPSIiPkZyb206PC9iPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0 ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj5DaHJpc3RvcGhlIFRSRUZPSVMgJmx0OzxhIGhyZWY9Im1h aWx0bzpjaHJpc3RvcGhlLnRyZWZvaXNAdW5pLmx1IiBjbGFzcz0iIj5jaHJpc3RvcGhlLnRyZWZv aXNAdW5pLmx1PC9hPiZndDs8YnIgY2xhc3M9IiI+DQo8YiBjbGFzcz0iIj5TZW50OjwvYj48c3Bh biBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+TWFyIDQsIDIwMTYg MjowOSBBTTxiciBjbGFzcz0iIj4NCjxiIGNsYXNzPSIiPlRvOjwvYj48c3BhbiBjbGFzcz0iQXBw bGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+UGF2ZWwgR2FzaGV2OzxzcGFuIGNsYXNz PSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJtYWlsdG86dXNl cnNAb3ZpcnQub3JnIiBjbGFzcz0iIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyIGNsYXNzPSIiPg0K PGIgY2xhc3M9IiI+U3ViamVjdDo8L2I+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFj ZSI+Jm5ic3A7PC9zcGFuPlJFOiBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBzdG9yYWdl PGJyIGNsYXNzPSIiPg0KPC9zcGFuPjwvZGl2Pg0KPC9kaXY+DQo8YnIgdHlwZT0iYXR0cmlidXRp b24iIGNsYXNzPSIiPg0KPC9kaXY+DQo8Zm9udCBzaXplPSIyIiBzdHlsZT0iZm9udC1mYW1pbHk6 IEhlbHZldGljYTsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6IG5vcm1hbDsgZm9u dC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgb3JwaGFuczogYXV0bzsg dGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25l OyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87IHdvcmQtc3BhY2luZzogMHB4OyAt d2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iZm9u dC1zaXplOiAxMHB0OyIgY2xhc3M9IiI+DQo8ZGl2IGNsYXNzPSJQbGFpblRleHQiPkhpLDxiciBj bGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NCldlIGhhdmUgYWxsIG5vZGVzIGxpa2UgeW91ICg1IGN1 cnJlbnRseSksIGFuZCBoYXZlIHRoZW0gc2V0dXAgYXMgTkZTIGluc3RlYWQgb2YgbG9jYWwgc3Rv cmFnZS48YnIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQpZb3UgaGF2ZSBzbGlnaHQgb3Zlcmhl YWQgYmVjYXVzZSBpdCdzIE5GUywgYnV0IG5vdCBzbyBtdWNoLCBhbmQgaXQgYWxsb3dzIHlvdSB0 byBtaWdyYXRlIFZNcyBhbmQgZGlza3MsIGV2ZW4gaWYgaXQncyBhIG1hbnVhbCBkaXNrLCBlZyBu byBIQSB3aWxsIGJlIHBvc3NpYmxlLjxiciBjbGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NClNvIHll YWgsIHRvIG1lLCBpdCdzIGEgZ29vZCBvcHRpb24gZ2l2ZW4geW91ciBzZXR1cC48YnIgY2xhc3M9 IiI+DQo8YnIgY2xhc3M9IiI+DQpCZXN0LDxiciBjbGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NCiZn dDsgLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS08YnIgY2xhc3M9IiI+DQomZ3Q7IEZyb206PHNw YW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Im1h aWx0bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNlcnMtYm91bmNlc0Bvdmly dC5vcmc8L2E+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFu Pls8YSBocmVmPSJtYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmciIGNsYXNzPSIiPm1haWx0 bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzwvYT5dDQogT24gQmVoYWxmPGJyIGNsYXNzPSIiPg0K Jmd0OyBPZiBQYXZlbCBHYXNoZXY8YnIgY2xhc3M9IiI+DQomZ3Q7IFNlbnQ6IGpldWRpIDMgbWFy cyAyMDE2IDIxOjQ0PGJyIGNsYXNzPSIiPg0KJmd0OyBUbzo8c3BhbiBjbGFzcz0iQXBwbGUtY29u dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0ibWFpbHRvOnVzZXJzQG92aXJ0Lm9y ZyIgY2xhc3M9IiI+dXNlcnNAb3ZpcnQub3JnPC9hPjxiciBjbGFzcz0iIj4NCiZndDsgU3ViamVj dDogW292aXJ0LXVzZXJzXSBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBzdG9yYWdlPGJy IGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNw Ozwvc3Bhbj48YnIgY2xhc3M9IiI+DQomZ3Q7IEhlbGxvLDxiciBjbGFzcz0iIj4NCiZndDs8c3Bh biBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIi Pg0KJmd0OyBJJ2QgbGlrZSB0byBhc2sgY29tbXVuaXR5LCB3aGF0IGlzIHRoZSBiZXN0IHdheSB0 byB1c2Ugb1ZpcnQgaW4gdGhlIGZvbGxvd2luZzxiciBjbGFzcz0iIj4NCiZndDsgaGFyZHdhcmUg Y29uZmlndXJhdGlvbjo8YnIgY2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZl cnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZndDsgVGhyZWUgc2VydmVy cyBjb25uZWN0ZWQgMUdCIG5ldHdvcmsuIEVhY2ggc2VydmVyIC0gMzIgdGhyZWFkcywgMjU2R0Ig UkFNLDxiciBjbGFzcz0iIj4NCiZndDsgNFRCIFJBSUQuPGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFu IGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+ DQomZ3Q7IFBsZWFzZSBub3RlIHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFuZCBhbiAxR0IgbmV0d29y ayBpcyBhIHR5cGljYWwgaGFyZHdhcmU8YnIgY2xhc3M9IiI+DQomZ3Q7IGNvbmZpZ3VyYXRpb24g Zm9yIGFsbW9zdCBhbnkgZGVkaWNhdGVkIGhvc3RpbmcuPGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFu IGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+ DQomZ3Q7IFVuZm9ydHVuYXRlbHksIG9WaXJ0IGRvZXNuJ3Qgc3VwcG9ydCBtdWx0aS1ub2RlIGxv Y2FsIHN0b3JhZ2UgY2x1c3RlcnMuPGJyIGNsYXNzPSIiPg0KJmd0OyBBbmQgR2x1c3Rlci9DRVBI IGRvZXNuJ3Qgd29yayB3ZWxsIG92ZXIgMUcgbmV0d29yay4gSXQgbG9va3MgbGlrZSB0aGF0IHRo ZTxiciBjbGFzcz0iIj4NCiZndDsgb25seSB3YXkgdG8gdXNlIG9WaXJ0IGluIGEgdGhyZWUtbm9k ZSBjbHVzdGVyIGlzIHRvIHNoYXJlIGxvY2FsIHN0b3JhZ2VzIG92ZXI8YnIgY2xhc3M9IiI+DQom Z3Q7IE5GUy4gQXQgbGVhc3QgaXQgbWFrZXMgcG9zc2libGUgdG8gbWlncmF0ZSBWTXMgYW5kIG1v dmUgZGlza3MgYW1vbmc8YnIgY2xhc3M9IiI+DQomZ3Q7IGhhcmR3YXJlIG5vZGVzLjxiciBjbGFz cz0iIj4NCiZndDs8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3Nw YW4+PGJyIGNsYXNzPSIiPg0KJmd0OyBEb2VzIHNvbWVib2R5IGhhdmUgc3VjaCBzZXR1cD88YnIg Y2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7 PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZndDsgVGhhbmtzPGJyIGNsYXNzPSIiPg0KJmd0OyBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxiciBjbGFzcz0iIj4N CiZndDsgVXNlcnMgbWFpbGluZyBsaXN0PGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJB cHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJtYWlsdG86VXNlcnNA b3ZpcnQub3JnIiBjbGFzcz0iIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyIGNsYXNzPSIiPg0KJmd0 OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVm PSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIGNsYXNzPSIi Pmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48L2Rpdj4N Cjwvc3Bhbj48L2ZvbnQ+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8L2Rpdj4NCjxiciBjbGFzcz0i Ij4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --_000_A1DB10E79F804D5A865F352A4C7ECE43unilu_--
--_000_A59A9C5DABA740C4BC588D04B4618567unilu_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgUGF2ZWwsDQoNCmVuZ2luZSBpcyBydW5uaW5nIG9uIG9uZSBvZiB0aGUgaG9zdHMuIFdlIGRv buKAmXQgaGF2ZSBIQSBmb3IgaG9zdCwgYnV0IEkgdGhpbmsgaXQgaXMgT0suDQoNCldlIHVzZWQg YSBkZWRpY2F0ZWQgTkZTIGZvciB0aGUgaG9zdGVkLWVuZ2luZSBvbiB0aGF0IGhvc3QuDQoNCkJl c3QsDQoNCkRyIENocmlzdG9waGUgVHJlZm9pcywgRGlwbC4tSW5nLg0KVGVjaG5pY2FsIFNwZWNp YWxpc3QgLyBQb3N0LURvYw0KDQpVTklWRVJTSVTDiSBEVSBMVVhFTUJPVVJHDQoNCkxVWEVNQk9V UkcgQ0VOVFJFIEZPUiBTWVNURU1TIEJJT01FRElDSU5FDQpDYW1wdXMgQmVsdmFsIHwgSG91c2Ug b2YgQmlvbWVkaWNpbmUNCjYsIGF2ZW51ZSBkdSBTd2luZw0KTC00MzY3IEJlbHZhdXgNClQ6ICsz NTIgNDYgNjYgNDQgNjEyNA0KRjogKzM1MiA0NiA2NiA0NCA2OTQ5DQpodHRwOi8vd3d3LnVuaS5s dS9sY3NiDQoNCltGYWNlYm9va108aHR0cHM6Ly93d3cuZmFjZWJvb2suY29tL3RyZWZleD4gIFtU d2l0dGVyXSA8aHR0cHM6Ly90d2l0dGVyLmNvbS9UcmVmZXg+ICAgW0dvb2dsZSBQbHVzXSA8aHR0 cHM6Ly9wbHVzLmdvb2dsZS5jb20vK0NocmlzdG9waGVUcmVmb2lzLz4gICBbTGlua2VkaW5dIDxo dHRwczovL3d3dy5saW5rZWRpbi5jb20vaW4vdHJlZm9pc2NocmlzdG9waGU+ICAgW3NreXBlXSA8 aHR0cDovL3NreXBlOlRyZWZleD9jYWxsPg0KDQotLS0tDQpUaGlzIG1lc3NhZ2UgaXMgY29uZmlk ZW50aWFsIGFuZCBtYXkgY29udGFpbiBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLg0KSXQgaXMgaW50 ZW5kZWQgZm9yIHRoZSBuYW1lZCByZWNpcGllbnQgb25seS4NCklmIHlvdSByZWNlaXZlIGl0IGlu IGVycm9yIHBsZWFzZSBub3RpZnkgbWUgYW5kIHBlcm1hbmVudGx5IGRlbGV0ZSB0aGUgb3JpZ2lu YWwgbWVzc2FnZSBhbmQgYW55IGNvcGllcy4NCi0tLS0NCg0KDQoNCk9uIDA0IE1hciAyMDE2LCBh dCAwNzo0NiwgUGF2ZWwgR2FzaGV2IDxQYXhAYWNyb25pcy5jb208bWFpbHRvOlBheEBhY3Jvbmlz LmNvbT4+IHdyb3RlOg0KDQoNCkNocmlzdG9waGUsDQoNCldoZXJlIGlzIHRoZSBlbmdpbmUgcnVu bmluZz8gRGlkIHlvdSB1c2Ugb25lIG9mIE5GU3MgZm9yIGhvc3RlZC1lbmdpbmU/DQoNClRoYW5r IHlvdQ0KDQoNCkZyb206IENocmlzdG9waGUgVFJFRk9JUyA8Y2hyaXN0b3BoZS50cmVmb2lzQHVu aS5sdTxtYWlsdG86Y2hyaXN0b3BoZS50cmVmb2lzQHVuaS5sdT4+DQpTZW50OiBNYXIgNCwgMjAx NiAyOjA5IEFNDQpUbzogUGF2ZWwgR2FzaGV2OyB1c2Vyc0BvdmlydC5vcmc8bWFpbHRvOnVzZXJz QG92aXJ0Lm9yZz4NClN1YmplY3Q6IFJFOiBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBz dG9yYWdlDQoNCkhpLA0KDQpXZSBoYXZlIGFsbCBub2RlcyBsaWtlIHlvdSAoNSBjdXJyZW50bHkp LCBhbmQgaGF2ZSB0aGVtIHNldHVwIGFzIE5GUyBpbnN0ZWFkIG9mIGxvY2FsIHN0b3JhZ2UuDQoN CllvdSBoYXZlIHNsaWdodCBvdmVyaGVhZCBiZWNhdXNlIGl0J3MgTkZTLCBidXQgbm90IHNvIG11 Y2gsIGFuZCBpdCBhbGxvd3MgeW91IHRvIG1pZ3JhdGUgVk1zIGFuZCBkaXNrcywgZXZlbiBpZiBp dCdzIGEgbWFudWFsIGRpc2ssIGVnIG5vIEhBIHdpbGwgYmUgcG9zc2libGUuDQoNClNvIHllYWgs IHRvIG1lLCBpdCdzIGEgZ29vZCBvcHRpb24gZ2l2ZW4geW91ciBzZXR1cC4NCg0KQmVzdCwNCg0K PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiB1c2Vycy1ib3VuY2VzQG92aXJ0 Lm9yZzxtYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmc+IFttYWlsdG86dXNlcnMtYm91bmNl c0BvdmlydC5vcmddIE9uIEJlaGFsZg0KPiBPZiBQYXZlbCBHYXNoZXYNCj4gU2VudDogamV1ZGkg MyBtYXJzIDIwMTYgMjE6NDQNCj4gVG86IHVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnNAb3Zp cnQub3JnPg0KPiBTdWJqZWN0OiBbb3ZpcnQtdXNlcnNdIE11bHRpLW5vZGUgY2x1c3RlciB3aXRo IGxvY2FsIHN0b3JhZ2UNCj4NCj4gSGVsbG8sDQo+DQo+IEknZCBsaWtlIHRvIGFzayBjb21tdW5p dHksIHdoYXQgaXMgdGhlIGJlc3Qgd2F5IHRvIHVzZSBvVmlydCBpbiB0aGUgZm9sbG93aW5nDQo+ IGhhcmR3YXJlIGNvbmZpZ3VyYXRpb246DQo+DQo+IFRocmVlIHNlcnZlcnMgY29ubmVjdGVkIDFH QiBuZXR3b3JrLiBFYWNoIHNlcnZlciAtIDMyIHRocmVhZHMsIDI1NkdCIFJBTSwNCj4gNFRCIFJB SUQuDQo+DQo+IFBsZWFzZSBub3RlIHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFuZCBhbiAxR0IgbmV0 d29yayBpcyBhIHR5cGljYWwgaGFyZHdhcmUNCj4gY29uZmlndXJhdGlvbiBmb3IgYWxtb3N0IGFu eSBkZWRpY2F0ZWQgaG9zdGluZy4NCj4NCj4gVW5mb3J0dW5hdGVseSwgb1ZpcnQgZG9lc24ndCBz dXBwb3J0IG11bHRpLW5vZGUgbG9jYWwgc3RvcmFnZSBjbHVzdGVycy4NCj4gQW5kIEdsdXN0ZXIv Q0VQSCBkb2Vzbid0IHdvcmsgd2VsbCBvdmVyIDFHIG5ldHdvcmsuIEl0IGxvb2tzIGxpa2UgdGhh dCB0aGUNCj4gb25seSB3YXkgdG8gdXNlIG9WaXJ0IGluIGEgdGhyZWUtbm9kZSBjbHVzdGVyIGlz IHRvIHNoYXJlIGxvY2FsIHN0b3JhZ2VzIG92ZXINCj4gTkZTLiBBdCBsZWFzdCBpdCBtYWtlcyBw b3NzaWJsZSB0byBtaWdyYXRlIFZNcyBhbmQgbW92ZSBkaXNrcyBhbW9uZw0KPiBoYXJkd2FyZSBu b2Rlcy4NCj4NCj4gRG9lcyBzb21lYm9keSBoYXZlIHN1Y2ggc2V0dXA/DQo+DQo+IFRoYW5rcw0K PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBVc2Vy cyBtYWlsaW5nIGxpc3QNCj4gVXNlcnNAb3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+ DQo+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycw0KDQo= --_000_A59A9C5DABA740C4BC588D04B4618567unilu_ Content-Type: text/html; charset="utf-8" Content-ID: <902AC57CAE7FDF4D969253BD26DE04FB@uni.lux> Content-Transfer-Encoding: base64 PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5IHN0eWxlPSJ3b3JkLXdy YXA6IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJy ZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPg0KSGkgUGF2ZWwsDQo8ZGl2IGNsYXNz PSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5lbmdpbmUgaXMgcnVubmlu ZyBvbiBvbmUgb2YgdGhlIGhvc3RzLiBXZSBkb27igJl0IGhhdmUgSEEgZm9yIGhvc3QsIGJ1dCBJ IHRoaW5rIGl0IGlzIE9LLjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+DQo8L2Rp dj4NCjxkaXYgY2xhc3M9IiI+V2UgdXNlZCBhIGRlZGljYXRlZCBORlMgZm9yIHRoZSBob3N0ZWQt ZW5naW5lIG9uIHRoYXQgaG9zdC48L2Rpdj4NCjxkaXYgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPg0K PC9kaXY+DQo8ZGl2IGNsYXNzPSIiPkJlc3QsPGJyIGNsYXNzPSIiPg0KPGRpdiBjbGFzcz0iIj4N CjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7 IG9ycGhhbnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0 LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lkb3dzOiBhdXRvOyB3b3Jk LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB3b3JkLXdyYXA6 IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJyZWFr OiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPg0KPHAgc3R5bGU9ImZvbnQtZmFtaWx5OiBB cmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMHB0OyBsaW5lLWhlaWdodDogMTZweDsgY29s b3I6IHJnYigzMywgMzMsIDMzKTsiIGNsYXNzPSIiPg0KPHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0 OiBib2xkOyBjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0i Ij5EciBDaHJpc3RvcGhlIFRyZWZvaXMsIERpcGwuLUluZy48L3NwYW4+PHNwYW4gY2xhc3M9IkFw cGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252 ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIg Y2xhc3M9IiI+PC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDYx LCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IGZvbnQtc2l6ZTogNy41cHQ7IiBjbGFzcz0iIj5U ZWNobmljYWwgU3BlY2lhbGlzdCAvIFBvc3QtRG9jPC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJmb250 LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogNy41cHQ7IGxpbmUtaGVpZ2h0 OiAxNnB4OyIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iZm9udC13ZWlnaHQ6IGJvbGQ7IGNvbG9y OiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPlVOSVZFUlNJVMOJ IERVIExVWEVNQk9VUkc8L3NwYW4+PGJyIGNsYXNzPSIiPg0KPHNwYW4gc3R5bGU9ImRpc3BsYXk6 IGlubGluZTsiIGNsYXNzPSIiPjxiciBjbGFzcz0iIj4NCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9u dC13ZWlnaHQ6IGJvbGQ7IGNvbG9yOiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsi IGNsYXNzPSIiPkxVWEVNQk9VUkcgQ0VOVFJFIEZPUiBTWVNURU1TIEJJT01FRElDSU5FPC9zcGFu PjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNw bGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5DYW1wdXMgQmVsdmFsIHwgSG91c2Ugb2YgQmlvbWVkaWNp bmU8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4g Y2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4N CjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPjYsIGF2ZW51ZSBkdSBTd2luZyZu YnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQpMLTQzNjcgQmVsdmF1eDxzcGFuIGNsYXNzPSJBcHBs ZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVy dGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxl PSJjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5UOjxz cGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48L3NwYW4+PHNw YW4gc3R5bGU9ImNvbG9yOiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNz PSIiPiYjNDM7MzUyIDQ2IDY2IDQ0IDYxMjQ8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZl cnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJjb2xv cjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5GOjxzcGFuIGNs YXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48L3NwYW4+PHNwYW4gc3R5 bGU9ImNvbG9yOiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPiYj NDM7MzUyIDQ2IDY2IDQ0IDY5NDk8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z cGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZu YnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGJyIGNs YXNzPSIiPg0KPC9zcGFuPjxhIGhyZWY9Imh0dHA6Ly93d3cudW5pLmx1L2xjc2IiIHN0eWxlPSJj b2xvcjogcmdiKDAsIDEwOSwgMTg5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+aHR0cDov L3d3dy51bmkubHUvbGNzYjwvYT48L3A+DQo8cCBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBz YW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7IGxpbmUtaGVpZ2h0OiAxNnB4OyIgY2xhc3M9IiI+ DQo8YSBocmVmPSJodHRwczovL3d3dy5mYWNlYm9vay5jb20vdHJlZmV4IiBzdHlsZT0iZGlzcGxh eTogaW5saW5lOyIgY2xhc3M9IiI+PGltZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmls ZW5hbWU9ImZhY2Vib29rLnBuZyIgc3JjPSJodHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNp Zy1hc3NldHMvcm91bmRlZC9mYWNlYm9vay5wbmciIGFsdD0iRmFjZWJvb2siIGNsYXNzPSIiPjwv YT48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4g Y2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Imh0dHBz Oi8vdHdpdHRlci5jb20vVHJlZmV4IiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+ PGltZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmlsZW5hbWU9InR3aXR0ZXIucG5nIiBz cmM9Imh0dHBzOi8vczMuYW1hem9uYXdzLmNvbS9odG1sc2lnLWFzc2V0cy9yb3VuZGVkL3R3aXR0 ZXIucG5nIiBhbHQ9IlR3aXR0ZXIiIGNsYXNzPSIiPjwvYT48c3BhbiBjbGFzcz0iQXBwbGUtY29u dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z cGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Imh0dHBzOi8vcGx1cy5nb29nbGUuY29tLyYjNDM7 Q2hyaXN0b3BoZVRyZWZvaXMvIiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGlt ZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmlsZW5hbWU9Imdvb2dsZXBsdXMucG5nIiBz cmM9Imh0dHBzOi8vczMuYW1hem9uYXdzLmNvbS9odG1sc2lnLWFzc2V0cy9yb3VuZGVkL2dvb2ds ZXBsdXMucG5nIiBhbHQ9Ikdvb2dsZSBQbHVzIiBjbGFzcz0iIj48L2E+PHNwYW4gY2xhc3M9IkFw cGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252 ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJodHRwczovL3d3dy5saW5rZWRpbi5j b20vaW4vdHJlZm9pc2NocmlzdG9waGUiIHN0eWxlPSJkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0i Ij48aW1nIHdpZHRoPSIyNCIgaGVpZ2h0PSIyNCIgZGF0YS1maWxlbmFtZT0ibGlua2VkaW4ucG5n IiBzcmM9Imh0dHBzOi8vczMuYW1hem9uYXdzLmNvbS9odG1sc2lnLWFzc2V0cy9yb3VuZGVkL2xp bmtlZGluLnBuZyIgYWx0PSJMaW5rZWRpbiIgY2xhc3M9IiI+PC9hPjxzcGFuIGNsYXNzPSJBcHBs ZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVy dGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0iaHR0cDovL3NreXBlOlRyZWZleD9jYWxs IiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGltZyB3aWR0aD0iMjQiIGhlaWdo dD0iMjQiIGRhdGEtZmlsZW5hbWU9InNreXBlLnBuZyIgc3JjPSJodHRwczovL3MzLmFtYXpvbmF3 cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC9za3lwZS5wbmciIGFsdD0ic2t5cGUiIGNsYXNz PSIiPjwvYT48L3A+DQo8cCBjbGFzcz0iYmFubmVyLWNvbnRhaW5lciIgc3R5bGU9ImZvbnQtZmFt aWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyBsaW5lLWhlaWdodDogMTZw eDsiPg0KPC9wPg0KPHAgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgY29s b3I6IHJnYig2MSwgNTksIDU5KTsgZm9udC1zaXplOiA5cHg7IGxpbmUtaGVpZ2h0OiAxNnB4OyIg Y2xhc3M9IiI+DQotLS0tPGJyIGNsYXNzPSIiPg0KVGhpcyBtZXNzYWdlIGlzIGNvbmZpZGVudGlh bCBhbmQgbWF5IGNvbnRhaW4gcHJpdmlsZWdlZCBpbmZvcm1hdGlvbi48c3BhbiBjbGFzcz0iQXBw bGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KSXQgaXMgaW50 ZW5kZWQgZm9yIHRoZSBuYW1lZCByZWNpcGllbnQgb25seS48c3BhbiBjbGFzcz0iQXBwbGUtY29u dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KSWYgeW91IHJlY2VpdmUg aXQgaW4gZXJyb3IgcGxlYXNlIG5vdGlmeSBtZSBhbmQgcGVybWFuZW50bHkgZGVsZXRlIHRoZSBv cmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkgY29waWVzLjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0 ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQotLS0tPGJyIGNsYXNzPSIiPg0K PC9wPg0KJm5ic3A7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9z cGFuPjwvZGl2Pg0KPC9kaXY+DQo8YnIgY2xhc3M9IiI+DQo8ZGl2Pg0KPGJsb2NrcXVvdGUgdHlw ZT0iY2l0ZSIgY2xhc3M9IiI+DQo8ZGl2IGNsYXNzPSIiPk9uIDA0IE1hciAyMDE2LCBhdCAwNzo0 NiwgUGF2ZWwgR2FzaGV2ICZsdDs8YSBocmVmPSJtYWlsdG86UGF4QGFjcm9uaXMuY29tIiBjbGFz cz0iIj5QYXhAYWNyb25pcy5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj4NCjxiciBjbGFzcz0iQXBw bGUtaW50ZXJjaGFuZ2UtbmV3bGluZSI+DQo8ZGl2IGNsYXNzPSIiPg0KPGRpdiBzdHlsZT0iZm9u dC1mYW1pbHk6IEhlbHZldGljYTsgZm9udC1zaXplOiAxNXB4OyBmb250LXN0eWxlOiBub3JtYWw7 IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGxldHRlci1z cGFjaW5nOiBub3JtYWw7IG9ycGhhbnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWlu ZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lk b3dzOiBhdXRvOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDog MHB4OyIgY2xhc3M9IiI+DQo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQ2FsaWJyaSwgQXJpYWws IEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0OyBjb2xvcjogcmdiKDMxLCA3 MywgMTI1KTsiIGNsYXNzPSIiPg0KPHAgZGlyPSJsdHIiIGNsYXNzPSIiPkNocmlzdG9waGUsPC9w Pg0KPHAgZGlyPSJsdHIiIGNsYXNzPSIiPldoZXJlIGlzIHRoZSBlbmdpbmUgcnVubmluZz8gRGlk IHlvdSB1c2Ugb25lIG9mIE5GU3MgZm9yIGhvc3RlZC1lbmdpbmU/PC9wPg0KPHAgZGlyPSJsdHIi IGNsYXNzPSIiPlRoYW5rIHlvdTwvcD4NCjxkaXYgaWQ9Inhfc2lnbmF0dXJlLXgiIHN0eWxlPSJm b250LWZhbWlseTogQ2FsaWJyaSwgQXJpYWwsIEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1z aXplOiAxMnB0OyBjb2xvcjogcmdiKDMxLCA3MywgMTI1KTsiIGNsYXNzPSIiPg0KPGJyIGNsYXNz PSIiPg0KPC9kaXY+DQo8L2Rpdj4NCjxkaXYgaWQ9InhfcXVvdGVkX2hlYWRlciIgc3R5bGU9ImNs ZWFyOiBib3RoOyIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPg0KPGRpdiBzdHlsZT0iYm9yZGVyLXN0 eWxlOiBzb2xpZCBub25lIG5vbmU7IGJvcmRlci10b3AtY29sb3I6IHJnYigyMjUsIDIyNSwgMjI1 KTsgYm9yZGVyLXRvcC13aWR0aDogMXB0OyBwYWRkaW5nOiAzcHQgMGNtIDBjbTsiIGNsYXNzPSIi Pg0KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFwdDsgZm9udC1mYW1pbHk6IENhbGlicmksIHNh bnMtc2VyaWY7IiBjbGFzcz0iIj48YiBjbGFzcz0iIj5Gcm9tOjwvYj48c3BhbiBjbGFzcz0iQXBw bGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+Q2hyaXN0b3BoZSBUUkVGT0lTICZsdDs8 YSBocmVmPSJtYWlsdG86Y2hyaXN0b3BoZS50cmVmb2lzQHVuaS5sdSIgY2xhc3M9IiI+Y2hyaXN0 b3BoZS50cmVmb2lzQHVuaS5sdTwvYT4mZ3Q7PGJyIGNsYXNzPSIiPg0KPGIgY2xhc3M9IiI+U2Vu dDo8L2I+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPk1h ciA0LCAyMDE2IDI6MDkgQU08YnIgY2xhc3M9IiI+DQo8YiBjbGFzcz0iIj5Ubzo8L2I+PHNwYW4g Y2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPlBhdmVsIEdhc2hldjs8 c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0i bWFpbHRvOnVzZXJzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNlcnNAb3ZpcnQub3JnPC9hPjxiciBj bGFzcz0iIj4NCjxiIGNsYXNzPSIiPlN1YmplY3Q6PC9iPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252 ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj5SRTogTXVsdGktbm9kZSBjbHVzdGVyIHdpdGggbG9j YWwgc3RvcmFnZTxiciBjbGFzcz0iIj4NCjwvc3Bhbj48L2Rpdj4NCjwvZGl2Pg0KPGJyIHR5cGU9 ImF0dHJpYnV0aW9uIiBjbGFzcz0iIj4NCjwvZGl2Pg0KPGZvbnQgc2l6ZT0iMiIgc3R5bGU9ImZv bnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNh cHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsg b3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQt dHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87IHdvcmQt c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IiBjbGFzcz0iIj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMHB0OyIgY2xhc3M9IiI+DQo8ZGl2IGNsYXNzPSJQbGFp blRleHQiPkhpLDxiciBjbGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NCldlIGhhdmUgYWxsIG5vZGVz IGxpa2UgeW91ICg1IGN1cnJlbnRseSksIGFuZCBoYXZlIHRoZW0gc2V0dXAgYXMgTkZTIGluc3Rl YWQgb2YgbG9jYWwgc3RvcmFnZS48YnIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQpZb3UgaGF2 ZSBzbGlnaHQgb3ZlcmhlYWQgYmVjYXVzZSBpdCdzIE5GUywgYnV0IG5vdCBzbyBtdWNoLCBhbmQg aXQgYWxsb3dzIHlvdSB0byBtaWdyYXRlIFZNcyBhbmQgZGlza3MsIGV2ZW4gaWYgaXQncyBhIG1h bnVhbCBkaXNrLCBlZyBubyBIQSB3aWxsIGJlIHBvc3NpYmxlLjxiciBjbGFzcz0iIj4NCjxiciBj bGFzcz0iIj4NClNvIHllYWgsIHRvIG1lLCBpdCdzIGEgZ29vZCBvcHRpb24gZ2l2ZW4geW91ciBz ZXR1cC48YnIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQpCZXN0LDxiciBjbGFzcz0iIj4NCjxi ciBjbGFzcz0iIj4NCiZndDsgLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS08YnIgY2xhc3M9IiI+ DQomZ3Q7IEZyb206PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9z cGFuPjxhIGhyZWY9Im1haWx0bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNl cnMtYm91bmNlc0BvdmlydC5vcmc8L2E+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFj ZSI+Jm5ic3A7PC9zcGFuPls8YSBocmVmPSJtYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmci IGNsYXNzPSIiPm1haWx0bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzwvYT5dDQogT24gQmVoYWxm PGJyIGNsYXNzPSIiPg0KJmd0OyBPZiBQYXZlbCBHYXNoZXY8YnIgY2xhc3M9IiI+DQomZ3Q7IFNl bnQ6IGpldWRpIDMgbWFycyAyMDE2IDIxOjQ0PGJyIGNsYXNzPSIiPg0KJmd0OyBUbzo8c3BhbiBj bGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0ibWFpbHRv OnVzZXJzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNlcnNAb3ZpcnQub3JnPC9hPjxiciBjbGFzcz0i Ij4NCiZndDsgU3ViamVjdDogW292aXJ0LXVzZXJzXSBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBs b2NhbCBzdG9yYWdlPGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0 ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQomZ3Q7IEhlbGxvLDxiciBjbGFz cz0iIj4NCiZndDs8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3Nw YW4+PGJyIGNsYXNzPSIiPg0KJmd0OyBJJ2QgbGlrZSB0byBhc2sgY29tbXVuaXR5LCB3aGF0IGlz IHRoZSBiZXN0IHdheSB0byB1c2Ugb1ZpcnQgaW4gdGhlIGZvbGxvd2luZzxiciBjbGFzcz0iIj4N CiZndDsgaGFyZHdhcmUgY29uZmlndXJhdGlvbjo8YnIgY2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xh c3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZn dDsgVGhyZWUgc2VydmVycyBjb25uZWN0ZWQgMUdCIG5ldHdvcmsuIEVhY2ggc2VydmVyIC0gMzIg dGhyZWFkcywgMjU2R0IgUkFNLDxiciBjbGFzcz0iIj4NCiZndDsgNFRCIFJBSUQuPGJyIGNsYXNz PSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bh bj48YnIgY2xhc3M9IiI+DQomZ3Q7IFBsZWFzZSBub3RlIHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFu ZCBhbiAxR0IgbmV0d29yayBpcyBhIHR5cGljYWwgaGFyZHdhcmU8YnIgY2xhc3M9IiI+DQomZ3Q7 IGNvbmZpZ3VyYXRpb24gZm9yIGFsbW9zdCBhbnkgZGVkaWNhdGVkIGhvc3RpbmcuPGJyIGNsYXNz PSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bh bj48YnIgY2xhc3M9IiI+DQomZ3Q7IFVuZm9ydHVuYXRlbHksIG9WaXJ0IGRvZXNuJ3Qgc3VwcG9y dCBtdWx0aS1ub2RlIGxvY2FsIHN0b3JhZ2UgY2x1c3RlcnMuPGJyIGNsYXNzPSIiPg0KJmd0OyBB bmQgR2x1c3Rlci9DRVBIIGRvZXNuJ3Qgd29yayB3ZWxsIG92ZXIgMUcgbmV0d29yay4gSXQgbG9v a3MgbGlrZSB0aGF0IHRoZTxiciBjbGFzcz0iIj4NCiZndDsgb25seSB3YXkgdG8gdXNlIG9WaXJ0 IGluIGEgdGhyZWUtbm9kZSBjbHVzdGVyIGlzIHRvIHNoYXJlIGxvY2FsIHN0b3JhZ2VzIG92ZXI8 YnIgY2xhc3M9IiI+DQomZ3Q7IE5GUy4gQXQgbGVhc3QgaXQgbWFrZXMgcG9zc2libGUgdG8gbWln cmF0ZSBWTXMgYW5kIG1vdmUgZGlza3MgYW1vbmc8YnIgY2xhc3M9IiI+DQomZ3Q7IGhhcmR3YXJl IG5vZGVzLjxiciBjbGFzcz0iIj4NCiZndDs8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNw YWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KJmd0OyBEb2VzIHNvbWVib2R5IGhhdmUg c3VjaCBzZXR1cD88YnIgY2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRl ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZndDsgVGhhbmtzPGJyIGNsYXNz PSIiPg0KJmd0OyBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XzxiciBjbGFzcz0iIj4NCiZndDsgVXNlcnMgbWFpbGluZyBsaXN0PGJyIGNsYXNzPSIiPg0KJmd0 OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVm PSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiBjbGFzcz0iIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJy IGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNw Ozwvc3Bhbj48YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8v dXNlcnMiIGNsYXNzPSIiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91 c2VyczwvYT48L2Rpdj4NCjwvc3Bhbj48L2ZvbnQ+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8L2Rp dj4NCjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K --_000_A59A9C5DABA740C4BC588D04B4618567unilu_--
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
Hello,
I'd like to ask community, what is the best way to use oVirt in the following hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB RAM, 4TB RAID.
Please note that a local storage and an 1GB network is a typical hardware configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes.
Do you know of reported problems with Gluster over 1Gb network? I think 10Gb is recommended, but 1Gb can also be used for gluster. (We use it in our lab setup, and haven't encountered any issues so far but of course, the workload may be different - hence the question)
Does somebody have such setup?
Thanks _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 04/03/16 12:22, "Sahina Bose" <sabose@redhat.com> wrote:
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes.
Do you know of reported problems with Gluster over 1Gb network? I think 10Gb is recommended, but 1Gb can also be used for gluster. (We use it in our lab setup, and haven't encountered any issues so far but of course, the workload may be different - hence the question)
Let's calculate. If I have a three node replicated gluster volume, each block writing on a node copies the block to the other two nodes. Thus, maximal write performance can't be above 50MB/s. Even it's acceptable for my workload, things get worse in failure recovering scenario. Gluster works with files. When a node fails and then recovers (even it's just a plain reboot), gluster copies the whole file over network if the file is changed during node outage. So if I have a 100GB VM disk, and guest system has written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain reboot of a node would result in datacenter out of service for several hours. Things might be better if you have a distributed+replicated gluster volume. It requires at least six nodes. But things are still bad when you try to rebalance the volume after adding new bricks, or when a node has really failed and replaced. Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
On 03/04/2016 04:13 PM, Pavel Gashev wrote:
On 04/03/16 12:22, "Sahina Bose" <sabose@redhat.com> wrote:
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes.
Do you know of reported problems with Gluster over 1Gb network? I think 10Gb is recommended, but 1Gb can also be used for gluster. (We use it in our lab setup, and haven't encountered any issues so far but of course, the workload may be different - hence the question) Let's calculate. If I have a three node replicated gluster volume, each block writing on a node copies the block to the other two nodes. Thus, maximal write performance can't be above 50MB/s. Even it's acceptable for my workload, things get worse in failure recovering scenario. Gluster works with files. When a node fails and then recovers (even it's just a plain reboot), gluster copies the whole file over network if the file is changed during node outage. So if I have a 100GB VM disk, and guest system has written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain reboot of a node would result in datacenter out of service for several hours.
Things might be better if you have a distributed+replicated gluster volume. It requires at least six nodes. But things are still bad when you try to rebalance the volume after adding new bricks, or when a node has really failed and replaced.
Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
Most of the problems that you outline here - related to healing and replacing are addressed with the sharding translator. Sharding breaks the large image file into smaller files, so that the entire file does not have to be copied. More details here - http://blog.gluster.org/2015/12/introducing-shard-translator/
On 04/03/16 13:50, "Sahina Bose" <sabose@redhat.com> wrote:
On 03/04/2016 04:13 PM, Pavel Gashev wrote:
On 04/03/16 12:22, "Sahina Bose" <sabose@redhat.com> wrote:
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes.
Do you know of reported problems with Gluster over 1Gb network? I think 10Gb is recommended, but 1Gb can also be used for gluster. (We use it in our lab setup, and haven't encountered any issues so far but of course, the workload may be different - hence the question) Let's calculate. If I have a three node replicated gluster volume, each block writing on a node copies the block to the other two nodes. Thus, maximal write performance can't be above 50MB/s. Even it's acceptable for my workload, things get worse in failure recovering scenario. Gluster works with files. When a node fails and then recovers (even it's just a plain reboot), gluster copies the whole file over network if the file is changed during node outage. So if I have a 100GB VM disk, and guest system has written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain reboot of a node would result in datacenter out of service for several hours.
Things might be better if you have a distributed+replicated gluster volume. It requires at least six nodes. But things are still bad when you try to rebalance the volume after adding new bricks, or when a node has really failed and replaced.
Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
Most of the problems that you outline here - related to healing and replacing are addressed with the sharding translator. Sharding breaks the large image file into smaller files, so that the entire file does not have to be copied. More details here - http://blog.gluster.org/2015/12/introducing-shard-translator/
Sure, I meant the same by mentioning distributed+replicated volumes. Actually, distributed+striped+replicated - https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Admin...
On 03/04/2016 05:30 PM, Pavel Gashev wrote:
On 04/03/16 13:50, "Sahina Bose" <sabose@redhat.com> wrote:
On 04/03/16 12:22, "Sahina Bose" <sabose@redhat.com> wrote:
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
Unfortunately, oVirt doesn't support multi-node local storage clusters. And Gluster/CEPH doesn't work well over 1G network. It looks like that the only way to use oVirt in a three-node cluster is to share local storages over NFS. At least it makes possible to migrate VMs and move disks among hardware nodes. Do you know of reported problems with Gluster over 1Gb network? I think 10Gb is recommended, but 1Gb can also be used for gluster. (We use it in our lab setup, and haven't encountered any issues so far but of course, the workload may be different - hence the question) Let's calculate. If I have a three node replicated gluster volume, each block writing on a node copies the block to the other two nodes. Thus, maximal write performance can't be above 50MB/s. Even it's acceptable for my workload, things get worse in failure recovering scenario. Gluster works with files. When a node fails and then recovers (even it's just a plain reboot), gluster copies the whole file over network if the file is changed during node outage. So if I have a 100GB VM disk, and guest system has written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain reboot of a node would result in datacenter out of service for several hours.
Things might be better if you have a distributed+replicated gluster volume. It requires at least six nodes. But things are still bad when you try to rebalance the volume after adding new bricks, or when a node has really failed and replaced.
Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO. Most of the problems that you outline here - related to healing and replacing are addressed with the sharding translator. Sharding breaks
On 03/04/2016 04:13 PM, Pavel Gashev wrote: the large image file into smaller files, so that the entire file does not have to be copied. More details here - http://blog.gluster.org/2015/12/introducing-shard-translator/ Sure, I meant the same by mentioning distributed+replicated volumes. Actually, distributed+striped+replicated - https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Admin...
Ok. Sharding is not the same as striped volumes in gluster. With striping, like you mentioned, you would require more number of nodes to form the striped set in addition to the replica set.( so 6 nodes since you need replica 3 ) Sharding can however work with 3 nodes - so on the replica 3 gluster volume that you create, you can turn on the volume option "features.shard on", to turn on this feature.
On 04/03/16 16:39, "Sahina Bose" <sabose@redhat.com> wrote:
On 03/04/2016 05:30 PM, Pavel Gashev wrote:
On 04/03/16 13:50, "Sahina Bose" <sabose@redhat.com> wrote:
Most of the problems that you outline here - related to healing and replacing are addressed with the sharding translator. Sharding breaks the large image file into smaller files, so that the entire file does not have to be copied. More details here - http://blog.gluster.org/2015/12/introducing-shard-translator/ Sure, I meant the same by mentioning distributed+replicated volumes. Actually, distributed+striped+replicated - https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Admin...
Ok. Sharding is not the same as striped volumes in gluster. With striping, like you mentioned, you would require more number of nodes to form the striped set in addition to the replica set.( so 6 nodes since you need replica 3 ) Sharding can however work with 3 nodes - so on the replica 3 gluster volume that you create, you can turn on the volume option "features.shard on", to turn on this feature.
Great feature. I think it must be "on" in oVirt by default.
participants (3)
-
Christophe TREFOIS -
Pavel Gashev -
Sahina Bose