11:44 p.m.
Hello,
I'd like to ask community, what is the best way to use oVirt in the
following hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB
RAM, 4TB RAID.
Please note that a local storage and an 1GB network is a typical
hardware configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters.
And Gluster/CEPH doesn't work well over 1G network. It looks like that
the only way to use oVirt in a three-node cluster is to share local
storages over NFS. At least it makes possible to migrate VMs and move
disks among hardware nodes.
Does somebody have such setup?
Thanks
2:09 a.m.
Hi,
We have all nodes like you (5 currently), and have them setup as NFS instead of local
storage.
You have slight overhead because it's NFS, but not so much, and it allows you to
migrate VMs and disks, even if it's a manual disk, eg no HA will be possible.
So yeah, to me, it's a good option given your setup.
Best,
-----Original Message-----
From: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] On Behalf
Of Pavel Gashev
Sent: jeudi 3 mars 2016 21:44
To: users(a)ovirt.org
Subject: [ovirt-users] Multi-node cluster with local storage
Hello,
I'd like to ask community, what is the best way to use oVirt in the following
hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB RAM,
4TB RAID.
Please note that a local storage and an 1GB network is a typical hardware
configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters.
And Gluster/CEPH doesn't work well over 1G network. It looks like that the
only way to use oVirt in a three-node cluster is to share local storages over
NFS. At least it makes possible to migrate VMs and move disks among
hardware nodes.
Does somebody have such setup?
Thanks
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:46 a.m.
--_000_2e54f1f30c004db3ab5621ff68a2ffeeacroniscom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Christophe,
Where is the engine running? Did you use one of NFSs for hosted-engine?
Thank you
From: Christophe TREFOIS <christophe.trefois(a)uni.lu Sent: Mar
4, 2016 2:09 AM
To: Pavel Gashev; users(a)ovirt.org
Subject: RE: Multi-node cluster with local storage
Hi,
We have all nodes like you (5 currently), and have them setup as NFS instea=
d of local storage.
You have slight overhead because it's NFS, but not so much, and it allows y=
ou to migrate VMs and disks, even if it's a manual disk, eg no HA will be p=
ossible.
So yeah, to me, it's a good option given your setup.
Best,
<head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft
Exchange Server" <!-- converted from text
--><style><!-- .EmailQuote { margin-left: 1pt; pad=
ding-left: 4pt; border-left: #800000 2px solid; } --></style </head
<body <style
type=3D"text/css" <!--
-- </style
<div <div
style=3D"font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.=
0pt; color:#1F497D" <p
dir=3D"ltr">Christophe,</p <p
dir=3D"ltr">Where is the engine running? Did you use one of NFSs for hos=
ted-engine?</p <p dir=3D"ltr">Thank
you</p <div
id=3D"x_signature-x" style=3D"font-family:Calibri,Arial,Helvetica,sans=
-serif; font-size:12.0pt; color:#1F497D"
<br </div </div <div
id=3D"x_quoted_header" style=3D"clear:both"><br <div style=3D"border:none; border-top:solid #E1E1E1
1.0pt; padding:3.0pt 0c=
m 0cm 0cm" <span
style=3D"font-size:11.0pt;
font-family:'Calibri','sans-serif'"><b>Fro=
m:</b> Christophe TREFOIS &lt;christophe.trefois(a)uni.lu&gt;<br <b>Sent:</b> Mar 4, 2016 2:09 AM<br <b>To:</b> Pavel Gashev; users(a)ovirt.org<br <b>Subject:</b> RE: Multi-node cluster with local
storage<br </span></div </div <br
type=3D"attribution" </div <font size=3D"2"><span
style=3D"font-size:10pt;" <div
class=3D"PlainText">Hi,<br
<br We have all nodes like you (5
currently), and have them setup as NFS instea=
d of local storage.<br <br You have
slight overhead because it's NFS, but not so much, and it allows y=
ou to migrate VMs and disks, even if it's a manual disk, eg no HA will be p=
ossible.<br <br So yeah,
to me, it's a good option given your setup.<br
<br Best,<br <br > -----Original
Message-----<br > From:
users-bounces(a)ovirt.org [<a href=3D"mailto:users-bounces@ovirt.o=
rg">mailto:users-bounces@ovirt.org</a>] On Behalf<br > Of Pavel Gashev<br >
Sent: jeudi 3 mars 2016 21:44<br >
To: users(a)ovirt.org<br > Subject: [ovirt-users]
Multi-node cluster with local storage<br >
<br > Hello,<br >
<br > I'd like to ask community, what is the best way to
use oVirt in the fo=
llowing<br > hardware
configuration:<br > <br > Three servers connected 1GB network. Each server - 32
threads, 256GB R=
AM,<br > 4TB RAID.<br > <br >
Please note that a local storage and an 1GB network is a typical hardw=
are<br > configuration for almost
any dedicated hosting.<br > <br > Unfortunately, oVirt doesn't support multi-node
local storage clusters=
.<br > And Gluster/CEPH doesn't work well over 1G
network. It looks like that=
the<br > only way to use oVirt in a
three-node cluster is to share local storag=
es over<br > NFS. At least it makes
possible to migrate VMs and move disks among<br=
> hardware nodes.<br >
<br > Does somebody have such setup?<br > <br >
Thanks<br >
_______________________________________________<br >
Users mailing list<br > Users(a)ovirt.org<br > <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.ovirt.org/mailman/listinfo/users</a><br
</div </span></font </body
</html
--_000_2e54f1f30c004db3ab5621ff68a2ffeeacroniscom_--
-----Original Message-----
From: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] On Behalf
Of Pavel Gashev
Sent: jeudi 3 mars 2016 21:44
To: users(a)ovirt.org
Subject: [ovirt-users] Multi-node cluster with local storage
Hello,
I'd like to ask community, what is the best way to use oVirt in the follo=
wing
hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB RAM,
4TB RAID.
Please note that a local storage and an 1GB network is a typical hardware
configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters.
And Gluster/CEPH doesn't work well over 1G network. It looks like that th=
e
only way to use oVirt in a three-node cluster is to share local
storages =
over
NFS. At least it makes possible to migrate VMs and move disks among
hardware nodes.
Does somebody have such setup?
Thanks
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--_000_2e54f1f30c004db3ab5621ff68a2ffeeacroniscom_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html
1:18 a.m.
--_000_A1DB10E79F804D5A865F352A4C7ECE43unilu_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
SGkgUGF2ZWwsDQoNCldlIHNldHVwIGFuIGV4dHJhIE5GUyBvbiBvbmUgb2YgdGhlIGhvc3RzLCBh
bmQgc2V0dXAgYXMgaG9zdGVkLWVuZ2luZS4NCg0KVGhhdCB3YXksIGl0IHdvcmtzIGZvciB1cy4N
Cg0KVXNpbmcgR2x1c3RlciAzLXdheSB3aXRoIENUREIgd2FzIG91ciBpbml0aWFsIHBsYW4sIGJ1
dCB3ZSBtb3ZlZCBhd2F5IGZyb20gaXQgYXMgaXQgc2VlbXMgdW5uZWNhc3NpcnkgY29tcGxpY2F0
ZWQuDQoNCktpbmQgcmVhZ3JkcywNCg0K4oCUDQpDaHJpc3RvcGhlDQoNCkRyIENocmlzdG9waGUg
VHJlZm9pcywgRGlwbC4tSW5nLg0KVGVjaG5pY2FsIFNwZWNpYWxpc3QgLyBQb3N0LURvYw0KDQpV
TklWRVJTSVTDiSBEVSBMVVhFTUJPVVJHDQoNCkxVWEVNQk9VUkcgQ0VOVFJFIEZPUiBTWVNURU1T
IEJJT01FRElDSU5FDQpDYW1wdXMgQmVsdmFsIHwgSG91c2Ugb2YgQmlvbWVkaWNpbmUNCjYsIGF2
ZW51ZSBkdSBTd2luZw0KTC00MzY3IEJlbHZhdXgNClQ6ICszNTIgNDYgNjYgNDQgNjEyNA0KRjog
KzM1MiA0NiA2NiA0NCA2OTQ5DQpodHRwOi8vd3d3LnVuaS5sdS9sY3NiDQoNCltGYWNlYm9va108
aHR0cHM6Ly93d3cuZmFjZWJvb2suY29tL3RyZWZleD4gIFtUd2l0dGVyXSA8aHR0cHM6Ly90d2l0
dGVyLmNvbS9UcmVmZXg+ICAgW0dvb2dsZSBQbHVzXSA8aHR0cHM6Ly9wbHVzLmdvb2dsZS5jb20v
K0NocmlzdG9waGVUcmVmb2lzLz4gICBbTGlua2VkaW5dIDxodHRwczovL3d3dy5saW5rZWRpbi5j
b20vaW4vdHJlZm9pc2NocmlzdG9waGU+ICAgW3NreXBlXSA8aHR0cDovL3NreXBlOlRyZWZleD9j
YWxsPg0KDQotLS0tDQpUaGlzIG1lc3NhZ2UgaXMgY29uZmlkZW50aWFsIGFuZCBtYXkgY29udGFp
biBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLg0KSXQgaXMgaW50ZW5kZWQgZm9yIHRoZSBuYW1lZCBy
ZWNpcGllbnQgb25seS4NCklmIHlvdSByZWNlaXZlIGl0IGluIGVycm9yIHBsZWFzZSBub3RpZnkg
bWUgYW5kIHBlcm1hbmVudGx5IGRlbGV0ZSB0aGUgb3JpZ2luYWwgbWVzc2FnZSBhbmQgYW55IGNv
cGllcy4NCi0tLS0NCg0KDQoNCk9uIDA0IE1hciAyMDE2LCBhdCAwNzo0NiwgUGF2ZWwgR2FzaGV2
IDxQYXhAYWNyb25pcy5jb208bWFpbHRvOlBheEBhY3JvbmlzLmNvbT4+IHdyb3RlOg0KDQoNCkNo
cmlzdG9waGUsDQoNCldoZXJlIGlzIHRoZSBlbmdpbmUgcnVubmluZz8gRGlkIHlvdSB1c2Ugb25l
IG9mIE5GU3MgZm9yIGhvc3RlZC1lbmdpbmU/DQoNClRoYW5rIHlvdQ0KDQoNCkZyb206IENocmlz
dG9waGUgVFJFRk9JUyA8Y2hyaXN0b3BoZS50cmVmb2lzQHVuaS5sdTxtYWlsdG86Y2hyaXN0b3Bo
ZS50cmVmb2lzQHVuaS5sdT4+DQpTZW50OiBNYXIgNCwgMjAxNiAyOjA5IEFNDQpUbzogUGF2ZWwg
R2FzaGV2OyB1c2Vyc0BvdmlydC5vcmc8bWFpbHRvOnVzZXJzQG92aXJ0Lm9yZz4NClN1YmplY3Q6
IFJFOiBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBzdG9yYWdlDQoNCkhpLA0KDQpXZSBo
YXZlIGFsbCBub2RlcyBsaWtlIHlvdSAoNSBjdXJyZW50bHkpLCBhbmQgaGF2ZSB0aGVtIHNldHVw
IGFzIE5GUyBpbnN0ZWFkIG9mIGxvY2FsIHN0b3JhZ2UuDQoNCllvdSBoYXZlIHNsaWdodCBvdmVy
aGVhZCBiZWNhdXNlIGl0J3MgTkZTLCBidXQgbm90IHNvIG11Y2gsIGFuZCBpdCBhbGxvd3MgeW91
IHRvIG1pZ3JhdGUgVk1zIGFuZCBkaXNrcywgZXZlbiBpZiBpdCdzIGEgbWFudWFsIGRpc2ssIGVn
IG5vIEhBIHdpbGwgYmUgcG9zc2libGUuDQoNClNvIHllYWgsIHRvIG1lLCBpdCdzIGEgZ29vZCBv
cHRpb24gZ2l2ZW4geW91ciBzZXR1cC4NCg0KQmVzdCwNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3Nh
Z2UtLS0tLQ0KPiBGcm9tOiB1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnMtYm91
bmNlc0BvdmlydC5vcmc+IFttYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmddIE9uIEJlaGFs
Zg0KPiBPZiBQYXZlbCBHYXNoZXYNCj4gU2VudDogamV1ZGkgMyBtYXJzIDIwMTYgMjE6NDQNCj4g
VG86IHVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnNAb3ZpcnQub3JnPg0KPiBTdWJqZWN0OiBb
b3ZpcnQtdXNlcnNdIE11bHRpLW5vZGUgY2x1c3RlciB3aXRoIGxvY2FsIHN0b3JhZ2UNCj4NCj4g
SGVsbG8sDQo+DQo+IEknZCBsaWtlIHRvIGFzayBjb21tdW5pdHksIHdoYXQgaXMgdGhlIGJlc3Qg
d2F5IHRvIHVzZSBvVmlydCBpbiB0aGUgZm9sbG93aW5nDQo+IGhhcmR3YXJlIGNvbmZpZ3VyYXRp
b246DQo+DQo+IFRocmVlIHNlcnZlcnMgY29ubmVjdGVkIDFHQiBuZXR3b3JrLiBFYWNoIHNlcnZl
ciAtIDMyIHRocmVhZHMsIDI1NkdCIFJBTSwNCj4gNFRCIFJBSUQuDQo+DQo+IFBsZWFzZSBub3Rl
IHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFuZCBhbiAxR0IgbmV0d29yayBpcyBhIHR5cGljYWwgaGFy
ZHdhcmUNCj4gY29uZmlndXJhdGlvbiBmb3IgYWxtb3N0IGFueSBkZWRpY2F0ZWQgaG9zdGluZy4N
Cj4NCj4gVW5mb3J0dW5hdGVseSwgb1ZpcnQgZG9lc24ndCBzdXBwb3J0IG11bHRpLW5vZGUgbG9j
YWwgc3RvcmFnZSBjbHVzdGVycy4NCj4gQW5kIEdsdXN0ZXIvQ0VQSCBkb2Vzbid0IHdvcmsgd2Vs
bCBvdmVyIDFHIG5ldHdvcmsuIEl0IGxvb2tzIGxpa2UgdGhhdCB0aGUNCj4gb25seSB3YXkgdG8g
dXNlIG9WaXJ0IGluIGEgdGhyZWUtbm9kZSBjbHVzdGVyIGlzIHRvIHNoYXJlIGxvY2FsIHN0b3Jh
Z2VzIG92ZXINCj4gTkZTLiBBdCBsZWFzdCBpdCBtYWtlcyBwb3NzaWJsZSB0byBtaWdyYXRlIFZN
cyBhbmQgbW92ZSBkaXNrcyBhbW9uZw0KPiBoYXJkd2FyZSBub2Rlcy4NCj4NCj4gRG9lcyBzb21l
Ym9keSBoYXZlIHN1Y2ggc2V0dXA/DQo+DQo+IFRoYW5rcw0KPiBfX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBVc2VycyBtYWlsaW5nIGxpc3QNCj4gVXNl
cnNAb3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+DQo+IGh0dHA6Ly9saXN0cy5vdmly
dC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycw0KDQo=
--_000_A1DB10E79F804D5A865F352A4C7ECE43unilu_
Content-Type: text/html; charset="utf-8"
Content-ID: <C647DAAEEBEFBF41BBCB33ED7B678FCD(a)uni.lux>
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5IHN0eWxlPSJ3b3JkLXdy
YXA6IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJy
ZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPg0KSGkgUGF2ZWwsDQo8ZGl2IGNsYXNz
PSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5XZSBzZXR1cCBhbiBleHRy
YSBORlMgb24gb25lIG9mIHRoZSBob3N0cywgYW5kIHNldHVwIGFzIGhvc3RlZC1lbmdpbmUuPC9k
aXY+DQo8ZGl2IGNsYXNzPSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5U
aGF0IHdheSwgaXQgd29ya3MgZm9yIHVzLjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9
IiI+DQo8L2Rpdj4NCjxkaXYgY2xhc3M9IiI+VXNpbmcgR2x1c3RlciAzLXdheSB3aXRoIENUREIg
d2FzIG91ciBpbml0aWFsIHBsYW4sIGJ1dCB3ZSBtb3ZlZCBhd2F5IGZyb20gaXQgYXMgaXQgc2Vl
bXMgdW5uZWNhc3NpcnkgY29tcGxpY2F0ZWQuPC9kaXY+DQo8ZGl2IGNsYXNzPSIiPjxiciBjbGFz
cz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5LaW5kIHJlYWdyZHMsPC9kaXY+DQo8ZGl2IGNs
YXNzPSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj7igJQ8L2Rpdj4NCjxk
aXYgY2xhc3M9IiI+Q2hyaXN0b3BoZTxiciBjbGFzcz0iIj4NCjxkaXYgY2xhc3M9IiI+DQo8ZGl2
IHN0eWxlPSJjb2xvcjogcmdiKDAsIDAsIDApOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBo
YW5zOiBhdXRvOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6IDBweDsgdGV4dC10cmFu
c2Zvcm06IG5vbmU7IHdoaXRlLXNwYWNlOiBub3JtYWw7IHdpZG93czogYXV0bzsgd29yZC1zcGFj
aW5nOiAwcHg7IC13ZWJraXQtdGV4dC1zdHJva2Utd2lkdGg6IDBweDsgd29yZC13cmFwOiBicmVh
ay13b3JkOyAtd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IC13ZWJraXQtbGluZS1icmVhazogYWZ0
ZXItd2hpdGUtc3BhY2U7IiBjbGFzcz0iIj4NCjxwIHN0eWxlPSJmb250LWZhbWlseTogQXJpYWws
IHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTBwdDsgbGluZS1oZWlnaHQ6IDE2cHg7IGNvbG9yOiBy
Z2IoMzMsIDMzLCAzMyk7IiBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJmb250LXdlaWdodDogYm9s
ZDsgY29sb3I6IHJnYig2MSwgNTksIDU5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+RHIg
Q2hyaXN0b3BoZSBUcmVmb2lzLCBEaXBsLi1JbmcuPC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1j
b252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVk
LXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNz
PSIiPjwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6IHJnYig2MSwgNTks
IDU5KTsgZGlzcGxheTogaW5saW5lOyBmb250LXNpemU6IDcuNXB0OyIgY2xhc3M9IiI+VGVjaG5p
Y2FsIFNwZWNpYWxpc3QgLyBQb3N0LURvYzwvc3Bhbj48L3A+DQo8cCBzdHlsZT0iZm9udC1mYW1p
bHk6IEFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDcuNXB0OyBsaW5lLWhlaWdodDogMTZw
eDsiIGNsYXNzPSIiPg0KPHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0OiBib2xkOyBjb2xvcjogcmdi
KDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5VTklWRVJTSVTDiSBEVSBM
VVhFTUJPVVJHPC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJkaXNwbGF5OiBpbmxp
bmU7IiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+DQo8L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQtd2Vp
Z2h0OiBib2xkOyBjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFz
cz0iIj5MVVhFTUJPVVJHIENFTlRSRSBGT1IgU1lTVEVNUyBCSU9NRURJQ0lORTwvc3Bhbj48YnIg
Y2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6IHJnYig2MSwgNTksIDU5KTsgZGlzcGxheTog
aW5saW5lOyIgY2xhc3M9IiI+Q2FtcHVzIEJlbHZhbCB8IEhvdXNlIG9mIEJpb21lZGljaW5lPHNw
YW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNz
PSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3Bh
biBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj42LCBhdmVudWUgZHUgU3dpbmcmbmJzcDs8
L3NwYW4+PGJyIGNsYXNzPSIiPg0KTC00MzY3IEJlbHZhdXg8c3BhbiBjbGFzcz0iQXBwbGUtY29u
dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z
cGFjZSI+Jm5ic3A7PC9zcGFuPjwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29s
b3I6IHJnYig2MSwgNTksIDU5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+VDo8c3BhbiBj
bGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxzcGFuIHN0
eWxlPSJjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj4m
IzQzOzM1MiA0NiA2NiA0NCA2MTI0PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQt
c3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iY29sb3I6IHJn
Yig2MSwgNTksIDU5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+Rjo8c3BhbiBjbGFzcz0i
QXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxzcGFuIHN0eWxlPSJj
b2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj4mIzQzOzM1
MiA0NiA2NiA0NCA2OTQ5PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2Ui
PiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8
L3NwYW4+PHNwYW4gc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxiciBjbGFzcz0i
Ij4NCjwvc3Bhbj48YSBocmVmPSJodHRwOi8vd3d3LnVuaS5sdS9sY3NiIiBzdHlsZT0iY29sb3I6
IHJnYigwLCAxMDksIDE4OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPmh0dHA6Ly93d3cu
dW5pLmx1L2xjc2I8L2E+PC9wPg0KPHAgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1z
ZXJpZjsgZm9udC1zaXplOiAxNHB4OyBsaW5lLWhlaWdodDogMTZweDsiIGNsYXNzPSIiPg0KPGEg
aHJlZj0iaHR0cHM6Ly93d3cuZmFjZWJvb2suY29tL3RyZWZleCIgc3R5bGU9ImRpc3BsYXk6IGlu
bGluZTsiIGNsYXNzPSIiPjxpbWcgd2lkdGg9IjI0IiBoZWlnaHQ9IjI0IiBkYXRhLWZpbGVuYW1l
PSJmYWNlYm9vay5wbmciIHNyYz0iaHR0cHM6Ly9zMy5hbWF6b25hd3MuY29tL2h0bWxzaWctYXNz
ZXRzL3JvdW5kZWQvZmFjZWJvb2sucG5nIiBhbHQ9IkZhY2Vib29rIiBjbGFzcz0iIj48L2E+PHNw
YW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNz
PSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJodHRwczovL3R3
aXR0ZXIuY29tL1RyZWZleCIgc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxpbWcg
d2lkdGg9IjI0IiBoZWlnaHQ9IjI0IiBkYXRhLWZpbGVuYW1lPSJ0d2l0dGVyLnBuZyIgc3JjPSJo
dHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC90d2l0dGVyLnBu
ZyIgYWx0PSJUd2l0dGVyIiBjbGFzcz0iIj48L2E+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRl
ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2Ui
PiZuYnNwOzwvc3Bhbj48YSBocmVmPSJodHRwczovL3BsdXMuZ29vZ2xlLmNvbS8mIzQzO0Nocmlz
dG9waGVUcmVmb2lzLyIgc3R5bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxpbWcgd2lk
dGg9IjI0IiBoZWlnaHQ9IjI0IiBkYXRhLWZpbGVuYW1lPSJnb29nbGVwbHVzLnBuZyIgc3JjPSJo
dHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC9nb29nbGVwbHVz
LnBuZyIgYWx0PSJHb29nbGUgUGx1cyIgY2xhc3M9IiI+PC9hPjxzcGFuIGNsYXNzPSJBcHBsZS1j
b252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVk
LXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0iaHR0cHM6Ly93d3cubGlua2VkaW4uY29tL2lu
L3RyZWZvaXNjaHJpc3RvcGhlIiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGlt
ZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmlsZW5hbWU9ImxpbmtlZGluLnBuZyIgc3Jj
PSJodHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC9saW5rZWRp
bi5wbmciIGFsdD0iTGlua2VkaW4iIGNsYXNzPSIiPjwvYT48c3BhbiBjbGFzcz0iQXBwbGUtY29u
dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z
cGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Imh0dHA6Ly9za3lwZTpUcmVmZXg/Y2FsbCIgc3R5
bGU9ImRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPjxpbWcgd2lkdGg9IjI0IiBoZWlnaHQ9IjI0
IiBkYXRhLWZpbGVuYW1lPSJza3lwZS5wbmciIHNyYz0iaHR0cHM6Ly9zMy5hbWF6b25hd3MuY29t
L2h0bWxzaWctYXNzZXRzL3JvdW5kZWQvc2t5cGUucG5nIiBhbHQ9InNreXBlIiBjbGFzcz0iIj48
L2E+PC9wPg0KPHAgY2xhc3M9ImJhbm5lci1jb250YWluZXIiIHN0eWxlPSJmb250LWZhbWlseTog
QXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogMTRweDsgbGluZS1oZWlnaHQ6IDE2cHg7Ij4N
CjwvcD4NCjxwIHN0eWxlPSJmb250LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGNvbG9yOiBy
Z2IoNjEsIDU5LCA1OSk7IGZvbnQtc2l6ZTogOXB4OyBsaW5lLWhlaWdodDogMTZweDsiIGNsYXNz
PSIiPg0KLS0tLTxiciBjbGFzcz0iIj4NClRoaXMgbWVzc2FnZSBpcyBjb25maWRlbnRpYWwgYW5k
IG1heSBjb250YWluIHByaXZpbGVnZWQgaW5mb3JtYXRpb24uPHNwYW4gY2xhc3M9IkFwcGxlLWNv
bnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCkl0IGlzIGludGVuZGVk
IGZvciB0aGUgbmFtZWQgcmVjaXBpZW50IG9ubHkuPHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRl
ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCklmIHlvdSByZWNlaXZlIGl0IGlu
IGVycm9yIHBsZWFzZSBub3RpZnkgbWUgYW5kIHBlcm1hbmVudGx5IGRlbGV0ZSB0aGUgb3JpZ2lu
YWwgbWVzc2FnZSBhbmQgYW55IGNvcGllcy48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNw
YWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KLS0tLTxiciBjbGFzcz0iIj4NCjwvcD4N
CiZuYnNwOzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48
L2Rpdj4NCjwvZGl2Pg0KPGJyIGNsYXNzPSIiPg0KPGRpdj4NCjxibG9ja3F1b3RlIHR5cGU9ImNp
dGUiIGNsYXNzPSIiPg0KPGRpdiBjbGFzcz0iIj5PbiAwNCBNYXIgMjAxNiwgYXQgMDc6NDYsIFBh
dmVsIEdhc2hldiAmbHQ7PGEgaHJlZj0ibWFpbHRvOlBheEBhY3JvbmlzLmNvbSIgY2xhc3M9IiI+
UGF4QGFjcm9uaXMuY29tPC9hPiZndDsgd3JvdGU6PC9kaXY+DQo8YnIgY2xhc3M9IkFwcGxlLWlu
dGVyY2hhbmdlLW5ld2xpbmUiPg0KPGRpdiBjbGFzcz0iIj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFt
aWx5OiBIZWx2ZXRpY2E7IGZvbnQtc2l6ZTogMTVweDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250
LXZhcmlhbnQ6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5v
cm1hbDsgb3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7
IHRleHQtdHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87
IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IiBjbGFz
cz0iIj4NCjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBDYWxpYnJpLCBBcmlhbCwgSGVsdmV0aWNh
LCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEycHQ7IGNvbG9yOiByZ2IoMzEsIDczLCAxMjUpOyIg
Y2xhc3M9IiI+DQo8cCBkaXI9Imx0ciIgY2xhc3M9IiI+Q2hyaXN0b3BoZSw8L3A+DQo8cCBkaXI9
Imx0ciIgY2xhc3M9IiI+V2hlcmUgaXMgdGhlIGVuZ2luZSBydW5uaW5nPyBEaWQgeW91IHVzZSBv
bmUgb2YgTkZTcyBmb3IgaG9zdGVkLWVuZ2luZT88L3A+DQo8cCBkaXI9Imx0ciIgY2xhc3M9IiI+
VGhhbmsgeW91PC9wPg0KPGRpdiBpZD0ieF9zaWduYXR1cmUteCIgc3R5bGU9ImZvbnQtZmFtaWx5
OiBDYWxpYnJpLCBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEycHQ7
IGNvbG9yOiByZ2IoMzEsIDczLCAxMjUpOyIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQo8L2Rp
dj4NCjwvZGl2Pg0KPGRpdiBpZD0ieF9xdW90ZWRfaGVhZGVyIiBzdHlsZT0iY2xlYXI6IGJvdGg7
IiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+DQo8ZGl2IHN0eWxlPSJib3JkZXItc3R5bGU6IHNvbGlk
IG5vbmUgbm9uZTsgYm9yZGVyLXRvcC1jb2xvcjogcmdiKDIyNSwgMjI1LCAyMjUpOyBib3JkZXIt
dG9wLXdpZHRoOiAxcHQ7IHBhZGRpbmc6IDNwdCAwY20gMGNtOyIgY2xhc3M9IiI+DQo8c3BhbiBz
dHlsZT0iZm9udC1zaXplOiAxMXB0OyBmb250LWZhbWlseTogQ2FsaWJyaSwgc2Fucy1zZXJpZjsi
IGNsYXNzPSIiPjxiIGNsYXNzPSIiPkZyb206PC9iPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0
ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj5DaHJpc3RvcGhlIFRSRUZPSVMgJmx0OzxhIGhyZWY9Im1h
aWx0bzpjaHJpc3RvcGhlLnRyZWZvaXNAdW5pLmx1IiBjbGFzcz0iIj5jaHJpc3RvcGhlLnRyZWZv
aXNAdW5pLmx1PC9hPiZndDs8YnIgY2xhc3M9IiI+DQo8YiBjbGFzcz0iIj5TZW50OjwvYj48c3Bh
biBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+TWFyIDQsIDIwMTYg
MjowOSBBTTxiciBjbGFzcz0iIj4NCjxiIGNsYXNzPSIiPlRvOjwvYj48c3BhbiBjbGFzcz0iQXBw
bGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+UGF2ZWwgR2FzaGV2OzxzcGFuIGNsYXNz
PSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJtYWlsdG86dXNl
cnNAb3ZpcnQub3JnIiBjbGFzcz0iIj51c2Vyc0BvdmlydC5vcmc8L2E+PGJyIGNsYXNzPSIiPg0K
PGIgY2xhc3M9IiI+U3ViamVjdDo8L2I+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFj
ZSI+Jm5ic3A7PC9zcGFuPlJFOiBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBzdG9yYWdl
PGJyIGNsYXNzPSIiPg0KPC9zcGFuPjwvZGl2Pg0KPC9kaXY+DQo8YnIgdHlwZT0iYXR0cmlidXRp
b24iIGNsYXNzPSIiPg0KPC9kaXY+DQo8Zm9udCBzaXplPSIyIiBzdHlsZT0iZm9udC1mYW1pbHk6
IEhlbHZldGljYTsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQ6IG5vcm1hbDsgZm9u
dC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsgb3JwaGFuczogYXV0bzsg
dGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQtdHJhbnNmb3JtOiBub25l
OyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87IHdvcmQtc3BhY2luZzogMHB4OyAt
d2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iZm9u
dC1zaXplOiAxMHB0OyIgY2xhc3M9IiI+DQo8ZGl2IGNsYXNzPSJQbGFpblRleHQiPkhpLDxiciBj
bGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NCldlIGhhdmUgYWxsIG5vZGVzIGxpa2UgeW91ICg1IGN1
cnJlbnRseSksIGFuZCBoYXZlIHRoZW0gc2V0dXAgYXMgTkZTIGluc3RlYWQgb2YgbG9jYWwgc3Rv
cmFnZS48YnIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQpZb3UgaGF2ZSBzbGlnaHQgb3Zlcmhl
YWQgYmVjYXVzZSBpdCdzIE5GUywgYnV0IG5vdCBzbyBtdWNoLCBhbmQgaXQgYWxsb3dzIHlvdSB0
byBtaWdyYXRlIFZNcyBhbmQgZGlza3MsIGV2ZW4gaWYgaXQncyBhIG1hbnVhbCBkaXNrLCBlZyBu
byBIQSB3aWxsIGJlIHBvc3NpYmxlLjxiciBjbGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NClNvIHll
YWgsIHRvIG1lLCBpdCdzIGEgZ29vZCBvcHRpb24gZ2l2ZW4geW91ciBzZXR1cC48YnIgY2xhc3M9
IiI+DQo8YnIgY2xhc3M9IiI+DQpCZXN0LDxiciBjbGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NCiZn
dDsgLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS08YnIgY2xhc3M9IiI+DQomZ3Q7IEZyb206PHNw
YW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Im1h
aWx0bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNlcnMtYm91bmNlc0Bvdmly
dC5vcmc8L2E+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFu
Pls8YSBocmVmPSJtYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmciIGNsYXNzPSIiPm1haWx0
bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzwvYT5dDQogT24gQmVoYWxmPGJyIGNsYXNzPSIiPg0K
Jmd0OyBPZiBQYXZlbCBHYXNoZXY8YnIgY2xhc3M9IiI+DQomZ3Q7IFNlbnQ6IGpldWRpIDMgbWFy
cyAyMDE2IDIxOjQ0PGJyIGNsYXNzPSIiPg0KJmd0OyBUbzo8c3BhbiBjbGFzcz0iQXBwbGUtY29u
dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0ibWFpbHRvOnVzZXJzQG92aXJ0Lm9y
ZyIgY2xhc3M9IiI+dXNlcnNAb3ZpcnQub3JnPC9hPjxiciBjbGFzcz0iIj4NCiZndDsgU3ViamVj
dDogW292aXJ0LXVzZXJzXSBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBzdG9yYWdlPGJy
IGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNw
Ozwvc3Bhbj48YnIgY2xhc3M9IiI+DQomZ3Q7IEhlbGxvLDxiciBjbGFzcz0iIj4NCiZndDs8c3Bh
biBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIi
Pg0KJmd0OyBJJ2QgbGlrZSB0byBhc2sgY29tbXVuaXR5LCB3aGF0IGlzIHRoZSBiZXN0IHdheSB0
byB1c2Ugb1ZpcnQgaW4gdGhlIGZvbGxvd2luZzxiciBjbGFzcz0iIj4NCiZndDsgaGFyZHdhcmUg
Y29uZmlndXJhdGlvbjo8YnIgY2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZl
cnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZndDsgVGhyZWUgc2VydmVy
cyBjb25uZWN0ZWQgMUdCIG5ldHdvcmsuIEVhY2ggc2VydmVyIC0gMzIgdGhyZWFkcywgMjU2R0Ig
UkFNLDxiciBjbGFzcz0iIj4NCiZndDsgNFRCIFJBSUQuPGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFu
IGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+
DQomZ3Q7IFBsZWFzZSBub3RlIHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFuZCBhbiAxR0IgbmV0d29y
ayBpcyBhIHR5cGljYWwgaGFyZHdhcmU8YnIgY2xhc3M9IiI+DQomZ3Q7IGNvbmZpZ3VyYXRpb24g
Zm9yIGFsbW9zdCBhbnkgZGVkaWNhdGVkIGhvc3RpbmcuPGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFu
IGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+
DQomZ3Q7IFVuZm9ydHVuYXRlbHksIG9WaXJ0IGRvZXNuJ3Qgc3VwcG9ydCBtdWx0aS1ub2RlIGxv
Y2FsIHN0b3JhZ2UgY2x1c3RlcnMuPGJyIGNsYXNzPSIiPg0KJmd0OyBBbmQgR2x1c3Rlci9DRVBI
IGRvZXNuJ3Qgd29yayB3ZWxsIG92ZXIgMUcgbmV0d29yay4gSXQgbG9va3MgbGlrZSB0aGF0IHRo
ZTxiciBjbGFzcz0iIj4NCiZndDsgb25seSB3YXkgdG8gdXNlIG9WaXJ0IGluIGEgdGhyZWUtbm9k
ZSBjbHVzdGVyIGlzIHRvIHNoYXJlIGxvY2FsIHN0b3JhZ2VzIG92ZXI8YnIgY2xhc3M9IiI+DQom
Z3Q7IE5GUy4gQXQgbGVhc3QgaXQgbWFrZXMgcG9zc2libGUgdG8gbWlncmF0ZSBWTXMgYW5kIG1v
dmUgZGlza3MgYW1vbmc8YnIgY2xhc3M9IiI+DQomZ3Q7IGhhcmR3YXJlIG5vZGVzLjxiciBjbGFz
cz0iIj4NCiZndDs8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3Nw
YW4+PGJyIGNsYXNzPSIiPg0KJmd0OyBEb2VzIHNvbWVib2R5IGhhdmUgc3VjaCBzZXR1cD88YnIg
Y2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7
PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZndDsgVGhhbmtzPGJyIGNsYXNzPSIiPg0KJmd0OyBfX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxiciBjbGFzcz0iIj4N
CiZndDsgVXNlcnMgbWFpbGluZyBsaXN0PGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJB
cHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJtYWlsdG86VXNlcnNA
b3ZpcnQub3JnIiBjbGFzcz0iIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyIGNsYXNzPSIiPg0KJmd0
OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVm
PSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIGNsYXNzPSIi
Pmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48L2Rpdj4N
Cjwvc3Bhbj48L2ZvbnQ+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8L2Rpdj4NCjxiciBjbGFzcz0i
Ij4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K
--_000_A1DB10E79F804D5A865F352A4C7ECE43unilu_--
2:17 p.m.
--_000_A59A9C5DABA740C4BC588D04B4618567unilu_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
SGkgUGF2ZWwsDQoNCmVuZ2luZSBpcyBydW5uaW5nIG9uIG9uZSBvZiB0aGUgaG9zdHMuIFdlIGRv
buKAmXQgaGF2ZSBIQSBmb3IgaG9zdCwgYnV0IEkgdGhpbmsgaXQgaXMgT0suDQoNCldlIHVzZWQg
YSBkZWRpY2F0ZWQgTkZTIGZvciB0aGUgaG9zdGVkLWVuZ2luZSBvbiB0aGF0IGhvc3QuDQoNCkJl
c3QsDQoNCkRyIENocmlzdG9waGUgVHJlZm9pcywgRGlwbC4tSW5nLg0KVGVjaG5pY2FsIFNwZWNp
YWxpc3QgLyBQb3N0LURvYw0KDQpVTklWRVJTSVTDiSBEVSBMVVhFTUJPVVJHDQoNCkxVWEVNQk9V
UkcgQ0VOVFJFIEZPUiBTWVNURU1TIEJJT01FRElDSU5FDQpDYW1wdXMgQmVsdmFsIHwgSG91c2Ug
b2YgQmlvbWVkaWNpbmUNCjYsIGF2ZW51ZSBkdSBTd2luZw0KTC00MzY3IEJlbHZhdXgNClQ6ICsz
NTIgNDYgNjYgNDQgNjEyNA0KRjogKzM1MiA0NiA2NiA0NCA2OTQ5DQpodHRwOi8vd3d3LnVuaS5s
dS9sY3NiDQoNCltGYWNlYm9va108aHR0cHM6Ly93d3cuZmFjZWJvb2suY29tL3RyZWZleD4gIFtU
d2l0dGVyXSA8aHR0cHM6Ly90d2l0dGVyLmNvbS9UcmVmZXg+ICAgW0dvb2dsZSBQbHVzXSA8aHR0
cHM6Ly9wbHVzLmdvb2dsZS5jb20vK0NocmlzdG9waGVUcmVmb2lzLz4gICBbTGlua2VkaW5dIDxo
dHRwczovL3d3dy5saW5rZWRpbi5jb20vaW4vdHJlZm9pc2NocmlzdG9waGU+ICAgW3NreXBlXSA8
aHR0cDovL3NreXBlOlRyZWZleD9jYWxsPg0KDQotLS0tDQpUaGlzIG1lc3NhZ2UgaXMgY29uZmlk
ZW50aWFsIGFuZCBtYXkgY29udGFpbiBwcml2aWxlZ2VkIGluZm9ybWF0aW9uLg0KSXQgaXMgaW50
ZW5kZWQgZm9yIHRoZSBuYW1lZCByZWNpcGllbnQgb25seS4NCklmIHlvdSByZWNlaXZlIGl0IGlu
IGVycm9yIHBsZWFzZSBub3RpZnkgbWUgYW5kIHBlcm1hbmVudGx5IGRlbGV0ZSB0aGUgb3JpZ2lu
YWwgbWVzc2FnZSBhbmQgYW55IGNvcGllcy4NCi0tLS0NCg0KDQoNCk9uIDA0IE1hciAyMDE2LCBh
dCAwNzo0NiwgUGF2ZWwgR2FzaGV2IDxQYXhAYWNyb25pcy5jb208bWFpbHRvOlBheEBhY3Jvbmlz
LmNvbT4+IHdyb3RlOg0KDQoNCkNocmlzdG9waGUsDQoNCldoZXJlIGlzIHRoZSBlbmdpbmUgcnVu
bmluZz8gRGlkIHlvdSB1c2Ugb25lIG9mIE5GU3MgZm9yIGhvc3RlZC1lbmdpbmU/DQoNClRoYW5r
IHlvdQ0KDQoNCkZyb206IENocmlzdG9waGUgVFJFRk9JUyA8Y2hyaXN0b3BoZS50cmVmb2lzQHVu
aS5sdTxtYWlsdG86Y2hyaXN0b3BoZS50cmVmb2lzQHVuaS5sdT4+DQpTZW50OiBNYXIgNCwgMjAx
NiAyOjA5IEFNDQpUbzogUGF2ZWwgR2FzaGV2OyB1c2Vyc0BvdmlydC5vcmc8bWFpbHRvOnVzZXJz
QG92aXJ0Lm9yZz4NClN1YmplY3Q6IFJFOiBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBsb2NhbCBz
dG9yYWdlDQoNCkhpLA0KDQpXZSBoYXZlIGFsbCBub2RlcyBsaWtlIHlvdSAoNSBjdXJyZW50bHkp
LCBhbmQgaGF2ZSB0aGVtIHNldHVwIGFzIE5GUyBpbnN0ZWFkIG9mIGxvY2FsIHN0b3JhZ2UuDQoN
CllvdSBoYXZlIHNsaWdodCBvdmVyaGVhZCBiZWNhdXNlIGl0J3MgTkZTLCBidXQgbm90IHNvIG11
Y2gsIGFuZCBpdCBhbGxvd3MgeW91IHRvIG1pZ3JhdGUgVk1zIGFuZCBkaXNrcywgZXZlbiBpZiBp
dCdzIGEgbWFudWFsIGRpc2ssIGVnIG5vIEhBIHdpbGwgYmUgcG9zc2libGUuDQoNClNvIHllYWgs
IHRvIG1lLCBpdCdzIGEgZ29vZCBvcHRpb24gZ2l2ZW4geW91ciBzZXR1cC4NCg0KQmVzdCwNCg0K
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiB1c2Vycy1ib3VuY2VzQG92aXJ0
Lm9yZzxtYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmc+IFttYWlsdG86dXNlcnMtYm91bmNl
c0BvdmlydC5vcmddIE9uIEJlaGFsZg0KPiBPZiBQYXZlbCBHYXNoZXYNCj4gU2VudDogamV1ZGkg
MyBtYXJzIDIwMTYgMjE6NDQNCj4gVG86IHVzZXJzQG92aXJ0Lm9yZzxtYWlsdG86dXNlcnNAb3Zp
cnQub3JnPg0KPiBTdWJqZWN0OiBbb3ZpcnQtdXNlcnNdIE11bHRpLW5vZGUgY2x1c3RlciB3aXRo
IGxvY2FsIHN0b3JhZ2UNCj4NCj4gSGVsbG8sDQo+DQo+IEknZCBsaWtlIHRvIGFzayBjb21tdW5p
dHksIHdoYXQgaXMgdGhlIGJlc3Qgd2F5IHRvIHVzZSBvVmlydCBpbiB0aGUgZm9sbG93aW5nDQo+
IGhhcmR3YXJlIGNvbmZpZ3VyYXRpb246DQo+DQo+IFRocmVlIHNlcnZlcnMgY29ubmVjdGVkIDFH
QiBuZXR3b3JrLiBFYWNoIHNlcnZlciAtIDMyIHRocmVhZHMsIDI1NkdCIFJBTSwNCj4gNFRCIFJB
SUQuDQo+DQo+IFBsZWFzZSBub3RlIHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFuZCBhbiAxR0IgbmV0
d29yayBpcyBhIHR5cGljYWwgaGFyZHdhcmUNCj4gY29uZmlndXJhdGlvbiBmb3IgYWxtb3N0IGFu
eSBkZWRpY2F0ZWQgaG9zdGluZy4NCj4NCj4gVW5mb3J0dW5hdGVseSwgb1ZpcnQgZG9lc24ndCBz
dXBwb3J0IG11bHRpLW5vZGUgbG9jYWwgc3RvcmFnZSBjbHVzdGVycy4NCj4gQW5kIEdsdXN0ZXIv
Q0VQSCBkb2Vzbid0IHdvcmsgd2VsbCBvdmVyIDFHIG5ldHdvcmsuIEl0IGxvb2tzIGxpa2UgdGhh
dCB0aGUNCj4gb25seSB3YXkgdG8gdXNlIG9WaXJ0IGluIGEgdGhyZWUtbm9kZSBjbHVzdGVyIGlz
IHRvIHNoYXJlIGxvY2FsIHN0b3JhZ2VzIG92ZXINCj4gTkZTLiBBdCBsZWFzdCBpdCBtYWtlcyBw
b3NzaWJsZSB0byBtaWdyYXRlIFZNcyBhbmQgbW92ZSBkaXNrcyBhbW9uZw0KPiBoYXJkd2FyZSBu
b2Rlcy4NCj4NCj4gRG9lcyBzb21lYm9keSBoYXZlIHN1Y2ggc2V0dXA/DQo+DQo+IFRoYW5rcw0K
PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBVc2Vy
cyBtYWlsaW5nIGxpc3QNCj4gVXNlcnNAb3ZpcnQub3JnPG1haWx0bzpVc2Vyc0BvdmlydC5vcmc+
DQo+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycw0KDQo=
--_000_A59A9C5DABA740C4BC588D04B4618567unilu_
Content-Type: text/html; charset="utf-8"
Content-ID: <902AC57CAE7FDF4D969253BD26DE04FB(a)uni.lux>
Content-Transfer-Encoding: base64
PGh0bWw+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0i
dGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjwvaGVhZD4NCjxib2R5IHN0eWxlPSJ3b3JkLXdy
YXA6IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJy
ZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPg0KSGkgUGF2ZWwsDQo8ZGl2IGNsYXNz
PSIiPjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj5lbmdpbmUgaXMgcnVubmlu
ZyBvbiBvbmUgb2YgdGhlIGhvc3RzLiBXZSBkb27igJl0IGhhdmUgSEEgZm9yIGhvc3QsIGJ1dCBJ
IHRoaW5rIGl0IGlzIE9LLjwvZGl2Pg0KPGRpdiBjbGFzcz0iIj48YnIgY2xhc3M9IiI+DQo8L2Rp
dj4NCjxkaXYgY2xhc3M9IiI+V2UgdXNlZCBhIGRlZGljYXRlZCBORlMgZm9yIHRoZSBob3N0ZWQt
ZW5naW5lIG9uIHRoYXQgaG9zdC48L2Rpdj4NCjxkaXYgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPg0K
PC9kaXY+DQo8ZGl2IGNsYXNzPSIiPkJlc3QsPGJyIGNsYXNzPSIiPg0KPGRpdiBjbGFzcz0iIj4N
CjxkaXYgc3R5bGU9ImNvbG9yOiByZ2IoMCwgMCwgMCk7IGxldHRlci1zcGFjaW5nOiBub3JtYWw7
IG9ycGhhbnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0
LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lkb3dzOiBhdXRvOyB3b3Jk
LXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDogMHB4OyB3b3JkLXdyYXA6
IGJyZWFrLXdvcmQ7IC13ZWJraXQtbmJzcC1tb2RlOiBzcGFjZTsgLXdlYmtpdC1saW5lLWJyZWFr
OiBhZnRlci13aGl0ZS1zcGFjZTsiIGNsYXNzPSIiPg0KPHAgc3R5bGU9ImZvbnQtZmFtaWx5OiBB
cmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMHB0OyBsaW5lLWhlaWdodDogMTZweDsgY29s
b3I6IHJnYigzMywgMzMsIDMzKTsiIGNsYXNzPSIiPg0KPHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0
OiBib2xkOyBjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0i
Ij5EciBDaHJpc3RvcGhlIFRyZWZvaXMsIERpcGwuLUluZy48L3NwYW4+PHNwYW4gY2xhc3M9IkFw
cGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252
ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIg
Y2xhc3M9IiI+PC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDYx
LCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IGZvbnQtc2l6ZTogNy41cHQ7IiBjbGFzcz0iIj5U
ZWNobmljYWwgU3BlY2lhbGlzdCAvIFBvc3QtRG9jPC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJmb250
LWZhbWlseTogQXJpYWwsIHNhbnMtc2VyaWY7IGZvbnQtc2l6ZTogNy41cHQ7IGxpbmUtaGVpZ2h0
OiAxNnB4OyIgY2xhc3M9IiI+DQo8c3BhbiBzdHlsZT0iZm9udC13ZWlnaHQ6IGJvbGQ7IGNvbG9y
OiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPlVOSVZFUlNJVMOJ
IERVIExVWEVNQk9VUkc8L3NwYW4+PGJyIGNsYXNzPSIiPg0KPHNwYW4gc3R5bGU9ImRpc3BsYXk6
IGlubGluZTsiIGNsYXNzPSIiPjxiciBjbGFzcz0iIj4NCjwvc3Bhbj48c3BhbiBzdHlsZT0iZm9u
dC13ZWlnaHQ6IGJvbGQ7IGNvbG9yOiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsi
IGNsYXNzPSIiPkxVWEVNQk9VUkcgQ0VOVFJFIEZPUiBTWVNURU1TIEJJT01FRElDSU5FPC9zcGFu
PjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNw
bGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5DYW1wdXMgQmVsdmFsIHwgSG91c2Ugb2YgQmlvbWVkaWNp
bmU8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4g
Y2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4N
CjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPjYsIGF2ZW51ZSBkdSBTd2luZyZu
YnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQpMLTQzNjcgQmVsdmF1eDxzcGFuIGNsYXNzPSJBcHBs
ZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVy
dGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxl
PSJjb2xvcjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5UOjxz
cGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48L3NwYW4+PHNw
YW4gc3R5bGU9ImNvbG9yOiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNz
PSIiPiYjNDM7MzUyIDQ2IDY2IDQ0IDYxMjQ8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZl
cnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCjxzcGFuIHN0eWxlPSJjb2xv
cjogcmdiKDYxLCA1OSwgNTkpOyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0iIj5GOjxzcGFuIGNs
YXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48L3NwYW4+PHNwYW4gc3R5
bGU9ImNvbG9yOiByZ2IoNjEsIDU5LCA1OSk7IGRpc3BsYXk6IGlubGluZTsiIGNsYXNzPSIiPiYj
NDM7MzUyIDQ2IDY2IDQ0IDY5NDk8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z
cGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZu
YnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGJyIGNs
YXNzPSIiPg0KPC9zcGFuPjxhIGhyZWY9Imh0dHA6Ly93d3cudW5pLmx1L2xjc2IiIHN0eWxlPSJj
b2xvcjogcmdiKDAsIDEwOSwgMTg5KTsgZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+aHR0cDov
L3d3dy51bmkubHUvbGNzYjwvYT48L3A+DQo8cCBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsLCBz
YW5zLXNlcmlmOyBmb250LXNpemU6IDE0cHg7IGxpbmUtaGVpZ2h0OiAxNnB4OyIgY2xhc3M9IiI+
DQo8YSBocmVmPSJodHRwczovL3d3dy5mYWNlYm9vay5jb20vdHJlZmV4IiBzdHlsZT0iZGlzcGxh
eTogaW5saW5lOyIgY2xhc3M9IiI+PGltZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmls
ZW5hbWU9ImZhY2Vib29rLnBuZyIgc3JjPSJodHRwczovL3MzLmFtYXpvbmF3cy5jb20vaHRtbHNp
Zy1hc3NldHMvcm91bmRlZC9mYWNlYm9vay5wbmciIGFsdD0iRmFjZWJvb2siIGNsYXNzPSIiPjwv
YT48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4g
Y2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Imh0dHBz
Oi8vdHdpdHRlci5jb20vVHJlZmV4IiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+
PGltZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmlsZW5hbWU9InR3aXR0ZXIucG5nIiBz
cmM9Imh0dHBzOi8vczMuYW1hem9uYXdzLmNvbS9odG1sc2lnLWFzc2V0cy9yb3VuZGVkL3R3aXR0
ZXIucG5nIiBhbHQ9IlR3aXR0ZXIiIGNsYXNzPSIiPjwvYT48c3BhbiBjbGFzcz0iQXBwbGUtY29u
dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1z
cGFjZSI+Jm5ic3A7PC9zcGFuPjxhIGhyZWY9Imh0dHBzOi8vcGx1cy5nb29nbGUuY29tLyYjNDM7
Q2hyaXN0b3BoZVRyZWZvaXMvIiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGlt
ZyB3aWR0aD0iMjQiIGhlaWdodD0iMjQiIGRhdGEtZmlsZW5hbWU9Imdvb2dsZXBsdXMucG5nIiBz
cmM9Imh0dHBzOi8vczMuYW1hem9uYXdzLmNvbS9odG1sc2lnLWFzc2V0cy9yb3VuZGVkL2dvb2ds
ZXBsdXMucG5nIiBhbHQ9Ikdvb2dsZSBQbHVzIiBjbGFzcz0iIj48L2E+PHNwYW4gY2xhc3M9IkFw
cGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252
ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVmPSJodHRwczovL3d3dy5saW5rZWRpbi5j
b20vaW4vdHJlZm9pc2NocmlzdG9waGUiIHN0eWxlPSJkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0i
Ij48aW1nIHdpZHRoPSIyNCIgaGVpZ2h0PSIyNCIgZGF0YS1maWxlbmFtZT0ibGlua2VkaW4ucG5n
IiBzcmM9Imh0dHBzOi8vczMuYW1hem9uYXdzLmNvbS9odG1sc2lnLWFzc2V0cy9yb3VuZGVkL2xp
bmtlZGluLnBuZyIgYWx0PSJMaW5rZWRpbiIgY2xhc3M9IiI+PC9hPjxzcGFuIGNsYXNzPSJBcHBs
ZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBjbGFzcz0iQXBwbGUtY29udmVy
dGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0iaHR0cDovL3NreXBlOlRyZWZleD9jYWxs
IiBzdHlsZT0iZGlzcGxheTogaW5saW5lOyIgY2xhc3M9IiI+PGltZyB3aWR0aD0iMjQiIGhlaWdo
dD0iMjQiIGRhdGEtZmlsZW5hbWU9InNreXBlLnBuZyIgc3JjPSJodHRwczovL3MzLmFtYXpvbmF3
cy5jb20vaHRtbHNpZy1hc3NldHMvcm91bmRlZC9za3lwZS5wbmciIGFsdD0ic2t5cGUiIGNsYXNz
PSIiPjwvYT48L3A+DQo8cCBjbGFzcz0iYmFubmVyLWNvbnRhaW5lciIgc3R5bGU9ImZvbnQtZmFt
aWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxNHB4OyBsaW5lLWhlaWdodDogMTZw
eDsiPg0KPC9wPg0KPHAgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbCwgc2Fucy1zZXJpZjsgY29s
b3I6IHJnYig2MSwgNTksIDU5KTsgZm9udC1zaXplOiA5cHg7IGxpbmUtaGVpZ2h0OiAxNnB4OyIg
Y2xhc3M9IiI+DQotLS0tPGJyIGNsYXNzPSIiPg0KVGhpcyBtZXNzYWdlIGlzIGNvbmZpZGVudGlh
bCBhbmQgbWF5IGNvbnRhaW4gcHJpdmlsZWdlZCBpbmZvcm1hdGlvbi48c3BhbiBjbGFzcz0iQXBw
bGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KSXQgaXMgaW50
ZW5kZWQgZm9yIHRoZSBuYW1lZCByZWNpcGllbnQgb25seS48c3BhbiBjbGFzcz0iQXBwbGUtY29u
dmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KSWYgeW91IHJlY2VpdmUg
aXQgaW4gZXJyb3IgcGxlYXNlIG5vdGlmeSBtZSBhbmQgcGVybWFuZW50bHkgZGVsZXRlIHRoZSBv
cmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkgY29waWVzLjxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0
ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQotLS0tPGJyIGNsYXNzPSIiPg0K
PC9wPg0KJm5ic3A7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9z
cGFuPjwvZGl2Pg0KPC9kaXY+DQo8YnIgY2xhc3M9IiI+DQo8ZGl2Pg0KPGJsb2NrcXVvdGUgdHlw
ZT0iY2l0ZSIgY2xhc3M9IiI+DQo8ZGl2IGNsYXNzPSIiPk9uIDA0IE1hciAyMDE2LCBhdCAwNzo0
NiwgUGF2ZWwgR2FzaGV2ICZsdDs8YSBocmVmPSJtYWlsdG86UGF4QGFjcm9uaXMuY29tIiBjbGFz
cz0iIj5QYXhAYWNyb25pcy5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj4NCjxiciBjbGFzcz0iQXBw
bGUtaW50ZXJjaGFuZ2UtbmV3bGluZSI+DQo8ZGl2IGNsYXNzPSIiPg0KPGRpdiBzdHlsZT0iZm9u
dC1mYW1pbHk6IEhlbHZldGljYTsgZm9udC1zaXplOiAxNXB4OyBmb250LXN0eWxlOiBub3JtYWw7
IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtd2VpZ2h0OiBub3JtYWw7IGxldHRlci1z
cGFjaW5nOiBub3JtYWw7IG9ycGhhbnM6IGF1dG87IHRleHQtYWxpZ246IHN0YXJ0OyB0ZXh0LWlu
ZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lk
b3dzOiBhdXRvOyB3b3JkLXNwYWNpbmc6IDBweDsgLXdlYmtpdC10ZXh0LXN0cm9rZS13aWR0aDog
MHB4OyIgY2xhc3M9IiI+DQo8ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQ2FsaWJyaSwgQXJpYWws
IEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMnB0OyBjb2xvcjogcmdiKDMxLCA3
MywgMTI1KTsiIGNsYXNzPSIiPg0KPHAgZGlyPSJsdHIiIGNsYXNzPSIiPkNocmlzdG9waGUsPC9w
Pg0KPHAgZGlyPSJsdHIiIGNsYXNzPSIiPldoZXJlIGlzIHRoZSBlbmdpbmUgcnVubmluZz8gRGlk
IHlvdSB1c2Ugb25lIG9mIE5GU3MgZm9yIGhvc3RlZC1lbmdpbmU/PC9wPg0KPHAgZGlyPSJsdHIi
IGNsYXNzPSIiPlRoYW5rIHlvdTwvcD4NCjxkaXYgaWQ9Inhfc2lnbmF0dXJlLXgiIHN0eWxlPSJm
b250LWZhbWlseTogQ2FsaWJyaSwgQXJpYWwsIEhlbHZldGljYSwgc2Fucy1zZXJpZjsgZm9udC1z
aXplOiAxMnB0OyBjb2xvcjogcmdiKDMxLCA3MywgMTI1KTsiIGNsYXNzPSIiPg0KPGJyIGNsYXNz
PSIiPg0KPC9kaXY+DQo8L2Rpdj4NCjxkaXYgaWQ9InhfcXVvdGVkX2hlYWRlciIgc3R5bGU9ImNs
ZWFyOiBib3RoOyIgY2xhc3M9IiI+PGJyIGNsYXNzPSIiPg0KPGRpdiBzdHlsZT0iYm9yZGVyLXN0
eWxlOiBzb2xpZCBub25lIG5vbmU7IGJvcmRlci10b3AtY29sb3I6IHJnYigyMjUsIDIyNSwgMjI1
KTsgYm9yZGVyLXRvcC13aWR0aDogMXB0OyBwYWRkaW5nOiAzcHQgMGNtIDBjbTsiIGNsYXNzPSIi
Pg0KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFwdDsgZm9udC1mYW1pbHk6IENhbGlicmksIHNh
bnMtc2VyaWY7IiBjbGFzcz0iIj48YiBjbGFzcz0iIj5Gcm9tOjwvYj48c3BhbiBjbGFzcz0iQXBw
bGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+Q2hyaXN0b3BoZSBUUkVGT0lTICZsdDs8
YSBocmVmPSJtYWlsdG86Y2hyaXN0b3BoZS50cmVmb2lzQHVuaS5sdSIgY2xhc3M9IiI+Y2hyaXN0
b3BoZS50cmVmb2lzQHVuaS5sdTwvYT4mZ3Q7PGJyIGNsYXNzPSIiPg0KPGIgY2xhc3M9IiI+U2Vu
dDo8L2I+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPk1h
ciA0LCAyMDE2IDI6MDkgQU08YnIgY2xhc3M9IiI+DQo8YiBjbGFzcz0iIj5Ubzo8L2I+PHNwYW4g
Y2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPlBhdmVsIEdhc2hldjs8
c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0i
bWFpbHRvOnVzZXJzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNlcnNAb3ZpcnQub3JnPC9hPjxiciBj
bGFzcz0iIj4NCjxiIGNsYXNzPSIiPlN1YmplY3Q6PC9iPjxzcGFuIGNsYXNzPSJBcHBsZS1jb252
ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj5SRTogTXVsdGktbm9kZSBjbHVzdGVyIHdpdGggbG9j
YWwgc3RvcmFnZTxiciBjbGFzcz0iIj4NCjwvc3Bhbj48L2Rpdj4NCjwvZGl2Pg0KPGJyIHR5cGU9
ImF0dHJpYnV0aW9uIiBjbGFzcz0iIj4NCjwvZGl2Pg0KPGZvbnQgc2l6ZT0iMiIgc3R5bGU9ImZv
bnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNh
cHM6IG5vcm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgbGV0dGVyLXNwYWNpbmc6IG5vcm1hbDsg
b3JwaGFuczogYXV0bzsgdGV4dC1hbGlnbjogc3RhcnQ7IHRleHQtaW5kZW50OiAwcHg7IHRleHQt
dHJhbnNmb3JtOiBub25lOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyB3aWRvd3M6IGF1dG87IHdvcmQt
c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7IiBjbGFzcz0iIj48
c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMHB0OyIgY2xhc3M9IiI+DQo8ZGl2IGNsYXNzPSJQbGFp
blRleHQiPkhpLDxiciBjbGFzcz0iIj4NCjxiciBjbGFzcz0iIj4NCldlIGhhdmUgYWxsIG5vZGVz
IGxpa2UgeW91ICg1IGN1cnJlbnRseSksIGFuZCBoYXZlIHRoZW0gc2V0dXAgYXMgTkZTIGluc3Rl
YWQgb2YgbG9jYWwgc3RvcmFnZS48YnIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQpZb3UgaGF2
ZSBzbGlnaHQgb3ZlcmhlYWQgYmVjYXVzZSBpdCdzIE5GUywgYnV0IG5vdCBzbyBtdWNoLCBhbmQg
aXQgYWxsb3dzIHlvdSB0byBtaWdyYXRlIFZNcyBhbmQgZGlza3MsIGV2ZW4gaWYgaXQncyBhIG1h
bnVhbCBkaXNrLCBlZyBubyBIQSB3aWxsIGJlIHBvc3NpYmxlLjxiciBjbGFzcz0iIj4NCjxiciBj
bGFzcz0iIj4NClNvIHllYWgsIHRvIG1lLCBpdCdzIGEgZ29vZCBvcHRpb24gZ2l2ZW4geW91ciBz
ZXR1cC48YnIgY2xhc3M9IiI+DQo8YnIgY2xhc3M9IiI+DQpCZXN0LDxiciBjbGFzcz0iIj4NCjxi
ciBjbGFzcz0iIj4NCiZndDsgLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS08YnIgY2xhc3M9IiI+
DQomZ3Q7IEZyb206PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9z
cGFuPjxhIGhyZWY9Im1haWx0bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNl
cnMtYm91bmNlc0BvdmlydC5vcmc8L2E+PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFj
ZSI+Jm5ic3A7PC9zcGFuPls8YSBocmVmPSJtYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5vcmci
IGNsYXNzPSIiPm1haWx0bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzwvYT5dDQogT24gQmVoYWxm
PGJyIGNsYXNzPSIiPg0KJmd0OyBPZiBQYXZlbCBHYXNoZXY8YnIgY2xhc3M9IiI+DQomZ3Q7IFNl
bnQ6IGpldWRpIDMgbWFycyAyMDE2IDIxOjQ0PGJyIGNsYXNzPSIiPg0KJmd0OyBUbzo8c3BhbiBj
bGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+PGEgaHJlZj0ibWFpbHRv
OnVzZXJzQG92aXJ0Lm9yZyIgY2xhc3M9IiI+dXNlcnNAb3ZpcnQub3JnPC9hPjxiciBjbGFzcz0i
Ij4NCiZndDsgU3ViamVjdDogW292aXJ0LXVzZXJzXSBNdWx0aS1ub2RlIGNsdXN0ZXIgd2l0aCBs
b2NhbCBzdG9yYWdlPGJyIGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0
ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YnIgY2xhc3M9IiI+DQomZ3Q7IEhlbGxvLDxiciBjbGFz
cz0iIj4NCiZndDs8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3Nw
YW4+PGJyIGNsYXNzPSIiPg0KJmd0OyBJJ2QgbGlrZSB0byBhc2sgY29tbXVuaXR5LCB3aGF0IGlz
IHRoZSBiZXN0IHdheSB0byB1c2Ugb1ZpcnQgaW4gdGhlIGZvbGxvd2luZzxiciBjbGFzcz0iIj4N
CiZndDsgaGFyZHdhcmUgY29uZmlndXJhdGlvbjo8YnIgY2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xh
c3M9IkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZn
dDsgVGhyZWUgc2VydmVycyBjb25uZWN0ZWQgMUdCIG5ldHdvcmsuIEVhY2ggc2VydmVyIC0gMzIg
dGhyZWFkcywgMjU2R0IgUkFNLDxiciBjbGFzcz0iIj4NCiZndDsgNFRCIFJBSUQuPGJyIGNsYXNz
PSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bh
bj48YnIgY2xhc3M9IiI+DQomZ3Q7IFBsZWFzZSBub3RlIHRoYXQgYSBsb2NhbCBzdG9yYWdlIGFu
ZCBhbiAxR0IgbmV0d29yayBpcyBhIHR5cGljYWwgaGFyZHdhcmU8YnIgY2xhc3M9IiI+DQomZ3Q7
IGNvbmZpZ3VyYXRpb24gZm9yIGFsbW9zdCBhbnkgZGVkaWNhdGVkIGhvc3RpbmcuPGJyIGNsYXNz
PSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bh
bj48YnIgY2xhc3M9IiI+DQomZ3Q7IFVuZm9ydHVuYXRlbHksIG9WaXJ0IGRvZXNuJ3Qgc3VwcG9y
dCBtdWx0aS1ub2RlIGxvY2FsIHN0b3JhZ2UgY2x1c3RlcnMuPGJyIGNsYXNzPSIiPg0KJmd0OyBB
bmQgR2x1c3Rlci9DRVBIIGRvZXNuJ3Qgd29yayB3ZWxsIG92ZXIgMUcgbmV0d29yay4gSXQgbG9v
a3MgbGlrZSB0aGF0IHRoZTxiciBjbGFzcz0iIj4NCiZndDsgb25seSB3YXkgdG8gdXNlIG9WaXJ0
IGluIGEgdGhyZWUtbm9kZSBjbHVzdGVyIGlzIHRvIHNoYXJlIGxvY2FsIHN0b3JhZ2VzIG92ZXI8
YnIgY2xhc3M9IiI+DQomZ3Q7IE5GUy4gQXQgbGVhc3QgaXQgbWFrZXMgcG9zc2libGUgdG8gbWln
cmF0ZSBWTXMgYW5kIG1vdmUgZGlza3MgYW1vbmc8YnIgY2xhc3M9IiI+DQomZ3Q7IGhhcmR3YXJl
IG5vZGVzLjxiciBjbGFzcz0iIj4NCiZndDs8c3BhbiBjbGFzcz0iQXBwbGUtY29udmVydGVkLXNw
YWNlIj4mbmJzcDs8L3NwYW4+PGJyIGNsYXNzPSIiPg0KJmd0OyBEb2VzIHNvbWVib2R5IGhhdmUg
c3VjaCBzZXR1cD88YnIgY2xhc3M9IiI+DQomZ3Q7PHNwYW4gY2xhc3M9IkFwcGxlLWNvbnZlcnRl
ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxiciBjbGFzcz0iIj4NCiZndDsgVGhhbmtzPGJyIGNsYXNz
PSIiPg0KJmd0OyBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
XzxiciBjbGFzcz0iIj4NCiZndDsgVXNlcnMgbWFpbGluZyBsaXN0PGJyIGNsYXNzPSIiPg0KJmd0
OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSBocmVm
PSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiBjbGFzcz0iIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJy
IGNsYXNzPSIiPg0KJmd0OzxzcGFuIGNsYXNzPSJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNw
Ozwvc3Bhbj48YSBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8v
dXNlcnMiIGNsYXNzPSIiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91
c2VyczwvYT48L2Rpdj4NCjwvc3Bhbj48L2ZvbnQ+PC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8L2Rp
dj4NCjxiciBjbGFzcz0iIj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K
--_000_A59A9C5DABA740C4BC588D04B4618567unilu_--
12:22 p.m.
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
Hello,
I'd like to ask community, what is the best way to use oVirt in the
following hardware configuration:
Three servers connected 1GB network. Each server - 32 threads, 256GB
RAM, 4TB RAID.
Please note that a local storage and an 1GB network is a typical
hardware configuration for almost any dedicated hosting.
Unfortunately, oVirt doesn't support multi-node local storage clusters.
And Gluster/CEPH doesn't work well over 1G network. It looks like that
the only way to use oVirt in a three-node cluster is to share local
storages over NFS. At least it makes possible to migrate VMs and move
disks among hardware nodes.
Do you know of reported problems with Gluster over 1Gb network? I think
10Gb is recommended, but 1Gb can also be used for gluster.
(We use it in our lab setup, and haven't encountered any issues so far
but of course, the workload may be different - hence the question)
Does somebody have such setup?
Thanks
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
1:43 p.m.
On 04/03/16 12:22, "Sahina Bose" <sabose(a)redhat.com> wrote:
On 03/04/2016 02:14 AM, Pavel Gashev wrote:
>
> Unfortunately, oVirt doesn't support multi-node local storage clusters.
> And Gluster/CEPH doesn't work well over 1G network. It looks like that
> the only way to use oVirt in a three-node cluster is to share local
> storages over NFS. At least it makes possible to migrate VMs and move
> disks among hardware nodes.
Do you know of reported problems with Gluster over 1Gb network? I think
10Gb is recommended, but 1Gb can also be used for gluster.
(We use it in our lab setup, and haven't encountered any issues so far
but of course, the workload may be different - hence the question)
Let's calculate. If I have a three node replicated gluster volume, each block writing
on a node copies the block to the other two nodes. Thus, maximal write performance
can't be above 50MB/s. Even it's acceptable for my workload, things get worse in
failure recovering scenario. Gluster works with files. When a node fails and then recovers
(even it's just a plain reboot), gluster copies the whole file over network if the
file is changed during node outage. So if I have a 100GB VM disk, and guest system has
written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It
might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during
recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain
reboot of a node would result in datacenter out of service for several hours.
Things might be better if you have a distributed+replicated gluster volume. It requires at
least six nodes. But things are still bad when you try to rebalance the volume after
adding new bricks, or when a node has really failed and replaced.
Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
1:50 p.m.
On 03/04/2016 04:13 PM, Pavel Gashev wrote:
On 04/03/16 12:22, "Sahina Bose" <sabose(a)redhat.com> wrote:
> On 03/04/2016 02:14 AM, Pavel Gashev wrote:
>> Unfortunately, oVirt doesn't support multi-node local storage clusters.
>> And Gluster/CEPH doesn't work well over 1G network. It looks like that
>> the only way to use oVirt in a three-node cluster is to share local
>> storages over NFS. At least it makes possible to migrate VMs and move
>> disks among hardware nodes.
>
> Do you know of reported problems with Gluster over 1Gb network? I think
> 10Gb is recommended, but 1Gb can also be used for gluster.
> (We use it in our lab setup, and haven't encountered any issues so far
> but of course, the workload may be different - hence the question)
Let's calculate. If I have a three node replicated gluster volume, each block writing
on a node copies the block to the other two nodes. Thus, maximal write performance
can't be above 50MB/s. Even it's acceptable for my workload, things get worse in
failure recovering scenario. Gluster works with files. When a node fails and then recovers
(even it's just a plain reboot), gluster copies the whole file over network if the
file is changed during node outage. So if I have a 100GB VM disk, and guest system has
written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It
might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during
recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain
reboot of a node would result in datacenter out of service for several hours.
Things might be better if you have a distributed+replicated gluster volume. It requires
at least six nodes. But things are still bad when you try to rebalance the volume after
adding new bricks, or when a node has really failed and replaced.
Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
Most of the problems that you outline here - related to healing and
replacing are addressed with the sharding translator. Sharding breaks
the large image file into smaller files, so that the entire file does
not have to be copied. More details here -
http://blog.gluster.org/2015/12/introducing-shard-translator/
3 p.m.
On 04/03/16 13:50, "Sahina Bose" <sabose(a)redhat.com> wrote:
On 03/04/2016 04:13 PM, Pavel Gashev wrote:
> On 04/03/16 12:22, "Sahina Bose" <sabose(a)redhat.com> wrote:
>> On 03/04/2016 02:14 AM, Pavel Gashev wrote:
>>> Unfortunately, oVirt doesn't support multi-node local storage clusters.
>>> And Gluster/CEPH doesn't work well over 1G network. It looks like that
>>> the only way to use oVirt in a three-node cluster is to share local
>>> storages over NFS. At least it makes possible to migrate VMs and move
>>> disks among hardware nodes.
>>
>> Do you know of reported problems with Gluster over 1Gb network? I think
>> 10Gb is recommended, but 1Gb can also be used for gluster.
>> (We use it in our lab setup, and haven't encountered any issues so far
>> but of course, the workload may be different - hence the question)
> Let's calculate. If I have a three node replicated gluster volume, each block
writing on a node copies the block to the other two nodes. Thus, maximal write performance
can't be above 50MB/s. Even it's acceptable for my workload, things get worse in
failure recovering scenario. Gluster works with files. When a node fails and then recovers
(even it's just a plain reboot), gluster copies the whole file over network if the
file is changed during node outage. So if I have a 100GB VM disk, and guest system has
written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It
might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during
recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain
reboot of a node would result in datacenter out of service for several hours.
>
> Things might be better if you have a distributed+replicated gluster volume. It
requires at least six nodes. But things are still bad when you try to rebalance the volume
after adding new bricks, or when a node has really failed and replaced.
>
> Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
Most of the problems that you outline here - related to healing and
replacing are addressed with the sharding translator. Sharding breaks
the large image file into smaller files, so that the entire file does
not have to be copied. More details here -
http://blog.gluster.org/2015/12/introducing-shard-translator/
Sure, I meant the same by mentioning distributed+replicated volumes. Actually,
distributed+striped+replicated -
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Ad...
4:39 p.m.
On 03/04/2016 05:30 PM, Pavel Gashev wrote:
On 04/03/16 13:50, "Sahina Bose" <sabose(a)redhat.com>
wrote:
> On 03/04/2016 04:13 PM, Pavel Gashev wrote:
>> On 04/03/16 12:22, "Sahina Bose" <sabose(a)redhat.com> wrote:
>>> On 03/04/2016 02:14 AM, Pavel Gashev wrote:
>>>> Unfortunately, oVirt doesn't support multi-node local storage
clusters.
>>>> And Gluster/CEPH doesn't work well over 1G network. It looks like
that
>>>> the only way to use oVirt in a three-node cluster is to share local
>>>> storages over NFS. At least it makes possible to migrate VMs and move
>>>> disks among hardware nodes.
>>> Do you know of reported problems with Gluster over 1Gb network? I think
>>> 10Gb is recommended, but 1Gb can also be used for gluster.
>>> (We use it in our lab setup, and haven't encountered any issues so far
>>> but of course, the workload may be different - hence the question)
>> Let's calculate. If I have a three node replicated gluster volume, each block
writing on a node copies the block to the other two nodes. Thus, maximal write performance
can't be above 50MB/s. Even it's acceptable for my workload, things get worse in
failure recovering scenario. Gluster works with files. When a node fails and then recovers
(even it's just a plain reboot), gluster copies the whole file over network if the
file is changed during node outage. So if I have a 100GB VM disk, and guest system has
written a 512-byte block to the disk, the whole 100GB will be copied during recovery. It
might take 20 minutes for 100GB, and 3 hours for 1TB. And network will be 100% busy during
recovery, so VMs on other nodes will wait for I/O most of time. In other words, a plain
reboot of a node would result in datacenter out of service for several hours.
>>
>> Things might be better if you have a distributed+replicated gluster volume. It
requires at least six nodes. But things are still bad when you try to rebalance the volume
after adding new bricks, or when a node has really failed and replaced.
>>
>> Thus, 1GB network is ok for a lab, but it's not ok for production. IMHO.
> Most of the problems that you outline here - related to healing and
> replacing are addressed with the sharding translator. Sharding breaks
> the large image file into smaller files, so that the entire file does
> not have to be copied. More details here -
> http://blog.gluster.org/2015/12/introducing-shard-translator/
Sure, I meant the same by mentioning distributed+replicated volumes. Actually,
distributed+striped+replicated -
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Ad...
Ok. Sharding is not the same as striped volumes in gluster. With
striping, like you mentioned, you would require more number of nodes to
form the striped set in addition to the replica set.( so 6 nodes since
you need replica 3 )
Sharding can however work with 3 nodes - so on the replica 3 gluster
volume that you create, you can turn on the volume option
"features.shard on", to turn on this feature.
5:53 p.m.
On 04/03/16 16:39, "Sahina Bose" <sabose(a)redhat.com> wrote:
On 03/04/2016 05:30 PM, Pavel Gashev wrote:
> On 04/03/16 13:50, "Sahina Bose" <sabose(a)redhat.com> wrote:
>> Most of the problems that you outline here - related to healing and
>> replacing are addressed with the sharding translator. Sharding breaks
>> the large image file into smaller files, so that the entire file does
>> not have to be copied. More details here -
>> http://blog.gluster.org/2015/12/introducing-shard-translator/
> Sure, I meant the same by mentioning distributed+replicated volumes. Actually,
distributed+striped+replicated -
https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Ad...
Ok. Sharding is not the same as striped volumes in gluster. With
striping, like you mentioned, you would require more number of nodes to
form the striped set in addition to the replica set.( so 6 nodes since
you need replica 3 )
Sharding can however work with 3 nodes - so on the replica 3 gluster
volume that you create, you can turn on the volume option
"features.shard on", to turn on this feature.
Great feature. I think it must be "on" in oVirt by default.
3145
days inactive
3185
days old
10 comments
3 participants
participants (3)
-
Christophe TREFOIS -
Pavel Gashev -
Sahina Bose