2:52 p.m.
----_com.android.email_1678282976805320
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64
CiAgICAKTmV0d29yayBwcm9ibGVtcyB3YXMgc29sdmVkIGFmdGVyIGNoYW5naW5nIEJvbmQgbW9k
ZSDCoChhbmQgaXQncyBzdHJhbmdlLiBJIGhhdmUgdG8gaW52ZXN0aWdhdGUgYXJvdW5kIHFlbXUt
a3ZtLCBjZW50byA3LjIgYW5kIHN3aXRjaCBmaXJtd2FyZSApLCBidXQgYnJva2VyIHByb2JsZW0g
c3RpbGwgZXhpc3QuIMKgSWYgSSB0dXJuIG9uIHRoZSBob3N0LCBoYSBhZ2VudCBzdGFydCBlbmdp
bmUgdm0uIFdoZW4gZW5naW5lIFZNIGlzIHVwLCBicm9rZXIgc3RyYXRzIHRvIHNlbmQgZW1haWwu
IMKgTm93IEkgaGF2ZW4ndCBoZXJlIGRldGFpbGVkIGxvZ3MuCgotLS0tLS0tLSBNZXNzYWdnaW8g
b3JpZ2luYWxlIC0tLS0tLS0tCkRhOiBZZWRpZHlhaCBCYXIgRGF2aWQgPGRpZGlAcmVkaGF0LmNv
bT4gCkRhdGE6IDIwLzEyLzIwMTUgIDExOjIwICAoR01UKzAxOjAwKSAKQTogU3RlZmFubyBEYW56
aSA8cy5kYW56aUBoYXdhaS5pdD4sIERhbiBLZW5pZ3NiZXJnIDxkYW5rZW5AcmVkaGF0LmNvbT4g
CkNjOiB1c2VycyA8dXNlcnNAb3ZpcnQub3JnPiAKT2dnZXR0bzogUmU6IFtvdmlydC11c2Vyc10g
TmV0d29yayBpbnN0YWJpbGl0eSBhZnRlciB1cGdyYWRlIDMuNi4wIC0+IDMuNi4xIAoKT24gRnJp
LCBEZWMgMTgsIDIwMTUgYXQgNTozMSBQTSwgU3RlZmFubyBEYW56aSA8cy5kYW56aUBoYXdhaS5p
dD4gd3JvdGU6Cj4gSSBmb3VuZCB0aGlzIGluIHZkc20ubG9nIGFuZCBJIHRoaW5rIHRoYXQgY291
bGQgYmUgdGhlIHByb2JsZW06Cj4KPiBUaHJlYWQtMzc3MTo6RVJST1I6OjIwMTUtMTItMTgKPiAx
NjoxODo1OCw1OTc6OmJyb2tlcmxpbms6OjI3OTo6b3ZpcnRfaG9zdGVkX2VuZ2luZV9oYS5saWIu
YnJva2VybGluay5Ccm9rZXJMaW5rOjooX2NvbW11bmljYXRlKQo+IENvbm5lY3Rpb24gY2xvc2Vk
OiBDb25uZWN0aW9uIGNsb3NlZAo+IFRocmVhZC0zNzcxOjpFUlJPUjo6MjAxNS0xMi0xOCAxNjox
ODo1OCw1OTc6OkFQSTo6MTg0Nzo6dmRzOjooX2dldEhhSW5mbykKPiBmYWlsZWQgdG8gcmV0cmll
dmUgSG9zdGVkIEVuZ2luZSBIQSBpbmZvCj4gVHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxh
c3QpOgo+wqDCoCBGaWxlICIvdXNyL3NoYXJlL3Zkc20vQVBJLnB5IiwgbGluZSAxODI3LCBpbiBf
Z2V0SGFJbmZvCj7CoMKgwqDCoCBzdGF0cyA9IGluc3RhbmNlLmdldF9hbGxfc3RhdHMoKQo+wqDC
oCBGaWxlCj4gIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0X2hvc3RlZF9l
bmdpbmVfaGEvY2xpZW50L2NsaWVudC5weSIsCj4gbGluZSAxMDMsIGluIGdldF9hbGxfc3RhdHMK
PsKgwqDCoMKgIHNlbGYuX2NvbmZpZ3VyZV9icm9rZXJfY29ubihicm9rZXIpCj7CoMKgIEZpbGUK
PiAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaG9zdGVkX2VuZ2luZV9o
YS9jbGllbnQvY2xpZW50LnB5IiwKPiBsaW5lIDE4MCwgaW4gX2NvbmZpZ3VyZV9icm9rZXJfY29u
bgo+wqDCoMKgwqAgZG9tX3R5cGU9ZG9tX3R5cGUpCj7CoMKgIEZpbGUKPiAiL3Vzci9saWIvcHl0
aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaG9zdGVkX2VuZ2luZV9oYS9saWIvYnJva2VybGlu
ay5weSIsCj4gbGluZSAxNzYsIGluIHNldF9zdG9yYWdlX2RvbWFpbgo+wqDCoMKgwqAgLmZvcm1h
dChzZF90eXBlLCBvcHRpb25zLCBlKSkKPiBSZXF1ZXN0RXJyb3I6IEZhaWxlZCB0byBzZXQgc3Rv
cmFnZSBkb21haW4gRmlsZXN5c3RlbUJhY2tlbmQsIG9wdGlvbnMKPiB7J2RvbV90eXBlJzogJ25m
czMnLCAnc2RfdXVpZCc6ICc0NmY1NWEzMS1mMzVmLTQ2NWMtYjNlMi1kZjQ1YzA1ZTA2YTcnfToK
PiBDb25uZWN0aW9uIGNsb3NlZAoKTXkgZ3Vlc3MgaXMgdGhhdCB0aGlzIGlzIGEgY29uc2VxdWVu
Y2Ugb2YgeW91ciBuZXR3b3JraW5nIHByb2JsZW1zLgoKQWRkaW5nIERhbi4KCj4KPgo+IElsIDE3
LzEyLzIwMTUgMTguNTEsIFN0ZWZhbm8gRGFuemkgaGEgc2NyaXR0bzoKPj4KPj4gSSBwYXJ0aWFs
bHkgc29sdmUgdGhlIHByb2JsZW0uCj4+Cj4+IE15IGhvc3QgbWFjaGluZSBoYXMgMiBuZXR3b3Jr
IGludGVyZmFjZXMgd2l0aCBhIGJvbmQuIFRoZSBib25kIHdhcwo+PiBjb25maWd1cmVkIHdpdGjC
oCBtb2RlPTQgKDgwMi4zYWQpIGFuZCBzd2l0Y2ggd2FzIGNvbmZpZ3VyZWQgaW4gdGhlIHNhbWUg
d2F5Lgo+PiBJZiBJIHJlbW92ZSBvbmUgbmV0d29yayBjYWJsZSB0aGUgbmV0d29yayBiZWNvbWUg
c3RhYmxlLiBXaXRoIGJvdGggY2FibGVzCj4+IGF0dGFjaGVkIHRoZSBuZXR3b3JrIGlzIGluc3Rh
YmxlLgo+Pgo+PiBJIHJlbW92ZWQgdGhlIGxpbmsgYWdncmVnYXRpb24gY29uZmlndXJhdGlvbiBm
cm9tIHN3aXRjaCBhbmQgY2hhbmdlIHRoZQo+PiBib25kIGluIG1vZGU9MiAoYmFsYW5jZS14b3Ip
LiBOb3cgdGhlIG5ldHdvcmsgYXJlIHN0YWJsZS4KPj4gVGhlIHN0cmFuZ2UgdGhpbmcgaXMgdGhh
dCBwcmV2aW91cyBjb25maWd1cmF0aW9uIHdvcmtlZCBmaW5lIGZvciBvbmUKPj4geWVhci4uLiBz
aW5jZSB0aGUgbGFzdCB1cGdyYWRlLgo+Pgo+PiBOb3cgaGEtYWdlbnQgZG9uJ3QgcmVib290IHRo
ZSBob3N0ZWQtZW5naW5lIGFueW1vcmUsIGJ1dCBJIHJlY2VpdmUgdHdvCj4+IGVtYWlscyBmcm9t
IGJyb2NrZXIgZXZlcmUgMi81IG1pbnV0ZXMuCj4+IEZpcnN0IGEgbWFpbCB3aXRoICJvdmlydC1o
b3N0ZWQtZW5naW5lIHN0YXRlIHRyYW5zaXRpb24KPj4gU3RhcnRTdGF0ZS1SZWluaXRpYWxpemVG
U00iIGFuZCBhZnRlciAib3ZpcnQtaG9zdGVkLWVuZ2luZSBzdGF0ZSB0cmFuc2l0aW9uCj4+IFJl
aW5pdGlhbGl6ZUZTTS1FbmdpbmVTdGFydGluZyIKPj4KPj4KPj4gSWwgMTcvMTIvMjAxNSAxMC41
MSwgU3RlZmFubyBEYW56aSBoYSBzY3JpdHRvOgo+Pj4KPj4+IEhlbGxvLAo+Pj4gSSBoYXZlIG9u
ZSB0ZXN0aW5nIGhvc3QgKG9ubHkgb25lIGhvc3QpIHdpdGggc2VsZiBob3N0ZWQgZW5naW5lIGFu
ZCAyIFZNCj4+PiAob25lIGxpbnV4IGFuZCBvbmUgd2luZG93cykuCj4+Pgo+Pj4gQWZ0ZXIgdXBn
cmFkZSBvdmlydCBmcm9tIDMuNi4wIHRvIDMuNi4xIHRoZSBuZXR3b3JrIGNvbm5lY3Rpb24gd29y
a3MKPj4+IGRpc2NvbnRpbnVvdXNseS4KPj4+IEV2ZXJ5IDEwIG1pbnV0ZXMgSEEgYWdlbnQgcmVz
dGFydCBob3N0ZWQgZW5naW5lIFZNIGJlY2F1c2UgcmVzdWx0IGRvd24uCj4+PiBCdXQgdGhlIG1h
Y2hpbmUgaXMgVVAsCj4+PiBvbmx5IHRoZSBuZXR3b3JrIHN0b3AgdG8gd29yayBmb3Igc29tZSBt
aW51dGVzLgo+Pj4gSSBhY3RpdmF0ZSBnbG9iYWwgbWFpbnRlbmFjZSBtb2RlIHRvIHByZXZlbnQg
ZW5naW5lIHJlYm9vdC4gSWYgSSBzc2ggdG8KPj4+IHRoZSBob3N0ZWQgZW5naW5lIHNvbWV0aW1l
cwo+Pj4gdGhlIGNvbm5lY3Rpb24gd29yayBhbmQgc29tZXRpbWVzIG5vLsKgIFVzaW5nIFZOQyBj
b25uZWN0aW9uIHRvIGVuZ2luZSBJCj4+PiBzZWUgdGhhdCBzb21ldGltZSBWTSByZWFjaCBleHRl
cm5hbCBuZXR3b3JrCj4+PiBhbmQgc29tZXRpbWVzIG5vLgo+Pj4gSWYgSSBkbyBhIHRjcGR1bXAg
b24gcGhpc2ljYWwgZXRoZXJuZXQgaW50ZXJmYWNlIEkgZG9uJ3Qgc2VlIGFueSBwYWNrZXQKPj4+
IHdoZW4gbmV0d29yayBvbiB2bSBkb24ndCB3b3JrLgo+Pj4KPj4+IFNhbWUgdGhpbmcgaGFwcGVu
cyBmbyBvdGhlcnMgdHdvIFZNLgo+Pj4KPj4+IEJlZm9yZSB0aGUgdXBncmFkZSBJIG5ldmVyIGhh
ZCBuZXR3b3JrIHByb2JsZW1zLgo+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX18KPj4+IFVzZXJzIG1haWxpbmcgbGlzdAo+Pj4gVXNlcnNAb3ZpcnQub3Jn
Cj4+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMKPj4+Cj4+
Cj4+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4+IFVz
ZXJzIG1haWxpbmcgbGlzdAo+PiBVc2Vyc0BvdmlydC5vcmcKPj4gaHR0cDovL2xpc3RzLm92aXJ0
Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzCj4KPiBfX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fXwo+IFVzZXJzIG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0
Lm9yZwo+IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2VycwoKCgot
LSAKRGlkaQo=
----_com.android.email_1678282976805320
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0
L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPjwvaGVhZD48Ym9keT4KICAgIAo8ZGl2Pk5ldHdvcmsgcHJv
YmxlbXMgd2FzIHNvbHZlZCBhZnRlciBjaGFuZ2luZyBCb25kIG1vZGUgJm5ic3A7KGFuZCBpdCdz
IHN0cmFuZ2UuIEkgaGF2ZSB0byBpbnZlc3RpZ2F0ZSBhcm91bmQgcWVtdS1rdm0sIGNlbnRvIDcu
MiBhbmQgc3dpdGNoIGZpcm13YXJlICksIGJ1dCBicm9rZXIgcHJvYmxlbSBzdGlsbCBleGlzdC4g
Jm5ic3A7SWYgSSB0dXJuIG9uIHRoZSBob3N0LCBoYSBhZ2VudCBzdGFydCBlbmdpbmUgdm0uIFdo
ZW4gZW5naW5lIFZNIGlzIHVwLCBicm9rZXIgc3RyYXRzIHRvIHNlbmQgZW1haWwuICZuYnNwO05v
dyBJIGhhdmVuJ3QgaGVyZSBkZXRhaWxlZCBsb2dzLjwvZGl2Pjxicj48YnI+LS0tLS0tLS0gTWVz
c2FnZ2lvIG9yaWdpbmFsZSAtLS0tLS0tLTxicj5EYTogWWVkaWR5YWggQmFyIERhdmlkICZsdDtk
aWRpQHJlZGhhdC5jb20mZ3Q7IDxicj5EYXRhOiAyMC8xMi8yMDE1ICAxMToyMCAgKEdNVCswMTow
MCkgPGJyPkE6IFN0ZWZhbm8gRGFuemkgJmx0O3MuZGFuemlAaGF3YWkuaXQmZ3Q7LCBEYW4gS2Vu
aWdzYmVyZyAmbHQ7ZGFua2VuQHJlZGhhdC5jb20mZ3Q7IDxicj5DYzogdXNlcnMgJmx0O3VzZXJz
QG92aXJ0Lm9yZyZndDsgPGJyPk9nZ2V0dG86IFJlOiBbb3ZpcnQtdXNlcnNdIE5ldHdvcmsgaW5z
dGFiaWxpdHkgYWZ0ZXIgdXBncmFkZSAzLjYuMCAtPiAzLjYuMSA8YnI+PGJyPk9uIEZyaSwgRGVj
IDE4LCAyMDE1IGF0IDU6MzEgUE0sIFN0ZWZhbm8gRGFuemkgJmx0O3MuZGFuemlAaGF3YWkuaXQm
Z3Q7IHdyb3RlOjxicj4mZ3Q7IEkgZm91bmQgdGhpcyBpbiB2ZHNtLmxvZyBhbmQgSSB0aGluayB0
aGF0IGNvdWxkIGJlIHRoZSBwcm9ibGVtOjxicj4mZ3Q7PGJyPiZndDsgVGhyZWFkLTM3NzE6OkVS
Uk9SOjoyMDE1LTEyLTE4PGJyPiZndDsgMTY6MTg6NTgsNTk3Ojpicm9rZXJsaW5rOjoyNzk6Om92
aXJ0X2hvc3RlZF9lbmdpbmVfaGEubGliLmJyb2tlcmxpbmsuQnJva2VyTGluazo6KF9jb21tdW5p
Y2F0ZSk8YnI+Jmd0OyBDb25uZWN0aW9uIGNsb3NlZDogQ29ubmVjdGlvbiBjbG9zZWQ8YnI+Jmd0
OyBUaHJlYWQtMzc3MTo6RVJST1I6OjIwMTUtMTItMTggMTY6MTg6NTgsNTk3OjpBUEk6OjE4NDc6
OnZkczo6KF9nZXRIYUluZm8pPGJyPiZndDsgZmFpbGVkIHRvIHJldHJpZXZlIEhvc3RlZCBFbmdp
bmUgSEEgaW5mbzxicj4mZ3Q7IFRyYWNlYmFjayAobW9zdCByZWNlbnQgY2FsbCBsYXN0KTo8YnI+
Jmd0OyZuYnNwOyZuYnNwOyBGaWxlICIvdXNyL3NoYXJlL3Zkc20vQVBJLnB5IiwgbGluZSAxODI3
LCBpbiBfZ2V0SGFJbmZvPGJyPiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgc3RhdHMgPSBp
bnN0YW5jZS5nZXRfYWxsX3N0YXRzKCk8YnI+Jmd0OyZuYnNwOyZuYnNwOyBGaWxlPGJyPiZndDsg
Ii91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL292aXJ0X2hvc3RlZF9lbmdpbmVfaGEv
Y2xpZW50L2NsaWVudC5weSIsPGJyPiZndDsgbGluZSAxMDMsIGluIGdldF9hbGxfc3RhdHM8YnI+
Jmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBzZWxmLl9jb25maWd1cmVfYnJva2VyX2Nvbm4o
YnJva2VyKTxicj4mZ3Q7Jm5ic3A7Jm5ic3A7IEZpbGU8YnI+Jmd0OyAiL3Vzci9saWIvcHl0aG9u
Mi43L3NpdGUtcGFja2FnZXMvb3ZpcnRfaG9zdGVkX2VuZ2luZV9oYS9jbGllbnQvY2xpZW50LnB5
Iiw8YnI+Jmd0OyBsaW5lIDE4MCwgaW4gX2NvbmZpZ3VyZV9icm9rZXJfY29ubjxicj4mZ3Q7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGRvbV90eXBlPWRvbV90eXBlKTxicj4mZ3Q7Jm5ic3A7Jm5i
c3A7IEZpbGU8YnI+Jmd0OyAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvb3ZpcnRf
aG9zdGVkX2VuZ2luZV9oYS9saWIvYnJva2VybGluay5weSIsPGJyPiZndDsgbGluZSAxNzYsIGlu
IHNldF9zdG9yYWdlX2RvbWFpbjxicj4mZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IC5mb3Jt
YXQoc2RfdHlwZSwgb3B0aW9ucywgZSkpPGJyPiZndDsgUmVxdWVzdEVycm9yOiBGYWlsZWQgdG8g
c2V0IHN0b3JhZ2UgZG9tYWluIEZpbGVzeXN0ZW1CYWNrZW5kLCBvcHRpb25zPGJyPiZndDsgeydk
b21fdHlwZSc6ICduZnMzJywgJ3NkX3V1aWQnOiAnNDZmNTVhMzEtZjM1Zi00NjVjLWIzZTItZGY0
NWMwNWUwNmE3J306PGJyPiZndDsgQ29ubmVjdGlvbiBjbG9zZWQ8YnI+PGJyPk15IGd1ZXNzIGlz
IHRoYXQgdGhpcyBpcyBhIGNvbnNlcXVlbmNlIG9mIHlvdXIgbmV0d29ya2luZyBwcm9ibGVtcy48
YnI+PGJyPkFkZGluZyBEYW4uPGJyPjxicj4mZ3Q7PGJyPiZndDs8YnI+Jmd0OyBJbCAxNy8xMi8y
MDE1IDE4LjUxLCBTdGVmYW5vIERhbnppIGhhIHNjcml0dG86PGJyPiZndDsmZ3Q7PGJyPiZndDsm
Z3Q7IEkgcGFydGlhbGx5IHNvbHZlIHRoZSBwcm9ibGVtLjxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0
OyBNeSBob3N0IG1hY2hpbmUgaGFzIDIgbmV0d29yayBpbnRlcmZhY2VzIHdpdGggYSBib25kLiBU
aGUgYm9uZCB3YXM8YnI+Jmd0OyZndDsgY29uZmlndXJlZCB3aXRoJm5ic3A7IG1vZGU9NCAoODAy
LjNhZCkgYW5kIHN3aXRjaCB3YXMgY29uZmlndXJlZCBpbiB0aGUgc2FtZSB3YXkuPGJyPiZndDsm
Z3Q7IElmIEkgcmVtb3ZlIG9uZSBuZXR3b3JrIGNhYmxlIHRoZSBuZXR3b3JrIGJlY29tZSBzdGFi
bGUuIFdpdGggYm90aCBjYWJsZXM8YnI+Jmd0OyZndDsgYXR0YWNoZWQgdGhlIG5ldHdvcmsgaXMg
aW5zdGFibGUuPGJyPiZndDsmZ3Q7PGJyPiZndDsmZ3Q7IEkgcmVtb3ZlZCB0aGUgbGluayBhZ2dy
ZWdhdGlvbiBjb25maWd1cmF0aW9uIGZyb20gc3dpdGNoIGFuZCBjaGFuZ2UgdGhlPGJyPiZndDsm
Z3Q7IGJvbmQgaW4gbW9kZT0yIChiYWxhbmNlLXhvcikuIE5vdyB0aGUgbmV0d29yayBhcmUgc3Rh
YmxlLjxicj4mZ3Q7Jmd0OyBUaGUgc3RyYW5nZSB0aGluZyBpcyB0aGF0IHByZXZpb3VzIGNvbmZp
Z3VyYXRpb24gd29ya2VkIGZpbmUgZm9yIG9uZTxicj4mZ3Q7Jmd0OyB5ZWFyLi4uIHNpbmNlIHRo
ZSBsYXN0IHVwZ3JhZGUuPGJyPiZndDsmZ3Q7PGJyPiZndDsmZ3Q7IE5vdyBoYS1hZ2VudCBkb24n
dCByZWJvb3QgdGhlIGhvc3RlZC1lbmdpbmUgYW55bW9yZSwgYnV0IEkgcmVjZWl2ZSB0d288YnI+
Jmd0OyZndDsgZW1haWxzIGZyb20gYnJvY2tlciBldmVyZSAyLzUgbWludXRlcy48YnI+Jmd0OyZn
dDsgRmlyc3QgYSBtYWlsIHdpdGggIm92aXJ0LWhvc3RlZC1lbmdpbmUgc3RhdGUgdHJhbnNpdGlv
bjxicj4mZ3Q7Jmd0OyBTdGFydFN0YXRlLVJlaW5pdGlhbGl6ZUZTTSIgYW5kIGFmdGVyICJvdmly
dC1ob3N0ZWQtZW5naW5lIHN0YXRlIHRyYW5zaXRpb248YnI+Jmd0OyZndDsgUmVpbml0aWFsaXpl
RlNNLUVuZ2luZVN0YXJ0aW5nIjxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0OyBJ
bCAxNy8xMi8yMDE1IDEwLjUxLCBTdGVmYW5vIERhbnppIGhhIHNjcml0dG86PGJyPiZndDsmZ3Q7
Jmd0Ozxicj4mZ3Q7Jmd0OyZndDsgSGVsbG8sPGJyPiZndDsmZ3Q7Jmd0OyBJIGhhdmUgb25lIHRl
c3RpbmcgaG9zdCAob25seSBvbmUgaG9zdCkgd2l0aCBzZWxmIGhvc3RlZCBlbmdpbmUgYW5kIDIg
Vk08YnI+Jmd0OyZndDsmZ3Q7IChvbmUgbGludXggYW5kIG9uZSB3aW5kb3dzKS48YnI+Jmd0OyZn
dDsmZ3Q7PGJyPiZndDsmZ3Q7Jmd0OyBBZnRlciB1cGdyYWRlIG92aXJ0IGZyb20gMy42LjAgdG8g
My42LjEgdGhlIG5ldHdvcmsgY29ubmVjdGlvbiB3b3Jrczxicj4mZ3Q7Jmd0OyZndDsgZGlzY29u
dGludW91c2x5Ljxicj4mZ3Q7Jmd0OyZndDsgRXZlcnkgMTAgbWludXRlcyBIQSBhZ2VudCByZXN0
YXJ0IGhvc3RlZCBlbmdpbmUgVk0gYmVjYXVzZSByZXN1bHQgZG93bi48YnI+Jmd0OyZndDsmZ3Q7
IEJ1dCB0aGUgbWFjaGluZSBpcyBVUCw8YnI+Jmd0OyZndDsmZ3Q7IG9ubHkgdGhlIG5ldHdvcmsg
c3RvcCB0byB3b3JrIGZvciBzb21lIG1pbnV0ZXMuPGJyPiZndDsmZ3Q7Jmd0OyBJIGFjdGl2YXRl
IGdsb2JhbCBtYWludGVuYWNlIG1vZGUgdG8gcHJldmVudCBlbmdpbmUgcmVib290LiBJZiBJIHNz
aCB0bzxicj4mZ3Q7Jmd0OyZndDsgdGhlIGhvc3RlZCBlbmdpbmUgc29tZXRpbWVzPGJyPiZndDsm
Z3Q7Jmd0OyB0aGUgY29ubmVjdGlvbiB3b3JrIGFuZCBzb21ldGltZXMgbm8uJm5ic3A7IFVzaW5n
IFZOQyBjb25uZWN0aW9uIHRvIGVuZ2luZSBJPGJyPiZndDsmZ3Q7Jmd0OyBzZWUgdGhhdCBzb21l
dGltZSBWTSByZWFjaCBleHRlcm5hbCBuZXR3b3JrPGJyPiZndDsmZ3Q7Jmd0OyBhbmQgc29tZXRp
bWVzIG5vLjxicj4mZ3Q7Jmd0OyZndDsgSWYgSSBkbyBhIHRjcGR1bXAgb24gcGhpc2ljYWwgZXRo
ZXJuZXQgaW50ZXJmYWNlIEkgZG9uJ3Qgc2VlIGFueSBwYWNrZXQ8YnI+Jmd0OyZndDsmZ3Q7IHdo
ZW4gbmV0d29yayBvbiB2bSBkb24ndCB3b3JrLjxicj4mZ3Q7Jmd0OyZndDs8YnI+Jmd0OyZndDsm
Z3Q7IFNhbWUgdGhpbmcgaGFwcGVucyBmbyBvdGhlcnMgdHdvIFZNLjxicj4mZ3Q7Jmd0OyZndDs8
YnI+Jmd0OyZndDsmZ3Q7IEJlZm9yZSB0aGUgdXBncmFkZSBJIG5ldmVyIGhhZCBuZXR3b3JrIHBy
b2JsZW1zLjxicj4mZ3Q7Jmd0OyZndDsgX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX188YnI+Jmd0OyZndDsmZ3Q7IFVzZXJzIG1haWxpbmcgbGlzdDxicj4mZ3Q7
Jmd0OyZndDsgVXNlcnNAb3ZpcnQub3JnPGJyPiZndDsmZ3Q7Jmd0OyBodHRwOi8vbGlzdHMub3Zp
cnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8YnI+Jmd0OyZndDsmZ3Q7PGJyPiZndDsmZ3Q7
PGJyPiZndDsmZ3Q7IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fPGJyPiZndDsmZ3Q7IFVzZXJzIG1haWxpbmcgbGlzdDxicj4mZ3Q7Jmd0OyBVc2Vyc0Bvdmly
dC5vcmc8YnI+Jmd0OyZndDsgaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZv
L3VzZXJzPGJyPiZndDs8YnI+Jmd0OyBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fXzxicj4mZ3Q7IFVzZXJzIG1haWxpbmcgbGlzdDxicj4mZ3Q7IFVzZXJzQG92
aXJ0Lm9yZzxicj4mZ3Q7IGh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91
c2Vyczxicj48YnI+PGJyPjxicj4tLSA8YnI+RGlkaTxicj48L2JvZHk+PC9odG1sPg==
----_com.android.email_1678282976805320--
3:06 p.m.
New subject: Network instability after upgrade 3.6.0 -> 3.6.1
On Sun, Dec 20, 2015 at 3:52 PM, Stefano Danzi <s.danzi(a)hawai.it> wrote:
Network problems was solved after changing Bond mode (and it's
strange. I
have to investigate around qemu-kvm, cento 7.2 and switch firmware ), but
broker problem still exist. If I turn on the host, ha agent start engine
vm. When engine VM is up, broker strats to send email. Now I haven't here
detailed logs.
Any updates? Please Cc with updates and logs also Martin (Cced).
Best,
--
Didi
2:48 p.m.
This is a multi-part message in MIME format.
--------------060609010302050806050207
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Problem solved!!!
The file hosted-engine.conf had a wrong fqdn.
I don't think that this happened during upgrade... mybe thay my
colleague did something of wrong...
Il 20/12/2015 14.52, Stefano Danzi ha scritto:
Network problems was solved after changing Bond mode (and it's
strange. I have to investigate around qemu-kvm, cento 7.2 and switch
firmware ), but broker problem still exist. If I turn on the host, ha
agent start engine vm. When engine VM is up, broker strats to send
email. Now I haven't here detailed logs.
-------- Messaggio originale --------
Da: Yedidyah Bar David <didi(a)redhat.com>
Data: 20/12/2015 11:20 (GMT+01:00)
A: Stefano Danzi <s.danzi(a)hawai.it>, Dan Kenigsberg <danken(a)redhat.com>
Cc: users <users(a)ovirt.org>
Oggetto: Re: [ovirt-users] Network instability after upgrade 3.6.0 ->
3.6.1
On Fri, Dec 18, 2015 at 5:31 PM, Stefano Danzi <s.danzi(a)hawai.it> wrote:
> I found this in vdsm.log and I think that could be the problem:
>
> Thread-3771::ERROR::2015-12-18
>
16:18:58,597::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate)
> Connection closed: Connection closed
> Thread-3771::ERROR::2015-12-18
16:18:58,597::API::1847::vds::(_getHaInfo)
> failed to retrieve Hosted Engine HA info
> Traceback (most recent call last):
> File "/usr/share/vdsm/API.py", line 1827, in _getHaInfo
> stats = instance.get_all_stats()
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 103, in get_all_stats
> self._configure_broker_conn(broker)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 180, in _configure_broker_conn
> dom_type=dom_type)
> File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 176, in set_storage_domain
> .format(sd_type, options, e))
> RequestError: Failed to set storage domain FilesystemBackend, options
> {'dom_type': 'nfs3', 'sd_uuid':
'46f55a31-f35f-465c-b3e2-df45c05e06a7'}:
> Connection closed
My guess is that this is a consequence of your networking problems.
Adding Dan.
>
>
> Il 17/12/2015 18.51, Stefano Danzi ha scritto:
>>
>> I partially solve the problem.
>>
>> My host machine has 2 network interfaces with a bond. The bond was
>> configured with mode=4 (802.3ad) and switch was configured in the
same way.
>> If I remove one network cable the network become stable. With both
cables
>> attached the network is instable.
>>
>> I removed the link aggregation configuration from switch and change the
>> bond in mode=2 (balance-xor). Now the network are stable.
>> The strange thing is that previous configuration worked fine for one
>> year... since the last upgrade.
>>
>> Now ha-agent don't reboot the hosted-engine anymore, but I receive two
>> emails from brocker evere 2/5 minutes.
>> First a mail with "ovirt-hosted-engine state transition
>> StartState-ReinitializeFSM" and after "ovirt-hosted-engine state
transition
>> ReinitializeFSM-EngineStarting"
>>
>>
>> Il 17/12/2015 10.51, Stefano Danzi ha scritto:
>>>
>>> Hello,
>>> I have one testing host (only one host) with self hosted engine
and 2 VM
>>> (one linux and one windows).
>>>
>>> After upgrade ovirt from 3.6.0 to 3.6.1 the network connection works
>>> discontinuously.
>>> Every 10 minutes HA agent restart hosted engine VM because result
down.
>>> But the machine is UP,
>>> only the network stop to work for some minutes.
>>> I activate global maintenace mode to prevent engine reboot. If I
ssh to
>>> the hosted engine sometimes
>>> the connection work and sometimes no. Using VNC connection to
engine I
>>> see that sometime VM reach external network
>>> and sometimes no.
>>> If I do a tcpdump on phisical ethernet interface I don't see any
packet
>>> when network on vm don't work.
>>>
>>> Same thing happens fo others two VM.
>>>
>>> Before the upgrade I never had network problems.
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Didi
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------060609010302050806050207
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Problem solved!!!<br>
<br>
The file hosted-engine.conf had a wrong fqdn.<br>
I don't think that this happened during upgrade... mybe thay my <span
id="result_box" class="short_text"
lang="en"><span class="hps">colleague
did something of wrong...</span></span><br>
<br>
<div class="moz-cite-prefix">Il 20/12/2015 14.52, Stefano Danzi ha
scritto:<br>
</div>
<blockquote
cite="mid:9iy7c4qvc9d7rykxr0v7k4h6.1450619291478@email.android.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div>Network problems was solved after changing Bond mode (and
it's strange. I have to investigate around qemu-kvm, cento 7.2
and switch firmware ), but broker problem still exist. If I
turn on the host, ha agent start engine vm. When engine VM is
up, broker strats to send email. Now I haven't here detailed
logs.</div>
<br>
<br>
-------- Messaggio originale --------<br>
Da: Yedidyah Bar David <a class="moz-txt-link-rfc2396E"
href="mailto:didi@redhat.com"><didi@redhat.com></a>
<br>
Data: 20/12/2015 11:20 (GMT+01:00) <br>
A: Stefano Danzi <a class="moz-txt-link-rfc2396E"
href="mailto:s.danzi@hawai.it"><s.danzi@hawai.it></a>,
Dan Kenigsberg
<a class="moz-txt-link-rfc2396E"
href="mailto:danken@redhat.com"><danken@redhat.com></a>
<br>
Cc: users <a class="moz-txt-link-rfc2396E"
href="mailto:users@ovirt.org"><users@ovirt.org></a>
<br>
Oggetto: Re: [ovirt-users] Network instability after upgrade 3.6.0
-> 3.6.1 <br>
<br>
On Fri, Dec 18, 2015 at 5:31 PM, Stefano Danzi
<a class="moz-txt-link-rfc2396E"
href="mailto:s.danzi@hawai.it"><s.danzi@hawai.it></a>
wrote:<br>
> I found this in vdsm.log and I think that could be the
problem:<br>
><br>
> Thread-3771::ERROR::2015-12-18<br>
>
16:18:58,597::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate)<br>
> Connection closed: Connection closed<br>
> Thread-3771::ERROR::2015-12-18
16:18:58,597::API::1847::vds::(_getHaInfo)<br>
> failed to retrieve Hosted Engine HA info<br>
> Traceback (most recent call last):<br>
> File "/usr/share/vdsm/API.py", line 1827, in
_getHaInfo<br>
> stats = instance.get_all_stats()<br>
> File<br>
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",<br>
> line 103, in get_all_stats<br>
> self._configure_broker_conn(broker)<br>
> File<br>
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",<br>
> line 180, in _configure_broker_conn<br>
> dom_type=dom_type)<br>
> File<br>
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",<br>
> line 176, in set_storage_domain<br>
> .format(sd_type, options, e))<br>
> RequestError: Failed to set storage domain FilesystemBackend,
options<br>
> {'dom_type': 'nfs3', 'sd_uuid':
'46f55a31-f35f-465c-b3e2-df45c05e06a7'}:<br>
> Connection closed<br>
<br>
My guess is that this is a consequence of your networking
problems.<br>
<br>
Adding Dan.<br>
<br>
><br>
><br>
> Il 17/12/2015 18.51, Stefano Danzi ha scritto:<br>
>><br>
>> I partially solve the problem.<br>
>><br>
>> My host machine has 2 network interfaces with a bond. The
bond was<br>
>> configured with mode=4 (802.3ad) and switch was
configured in the same way.<br>
>> If I remove one network cable the network become stable.
With both cables<br>
>> attached the network is instable.<br>
>><br>
>> I removed the link aggregation configuration from switch
and change the<br>
>> bond in mode=2 (balance-xor). Now the network are
stable.<br>
>> The strange thing is that previous configuration worked
fine for one<br>
>> year... since the last upgrade.<br>
>><br>
>> Now ha-agent don't reboot the hosted-engine anymore, but
I receive two<br>
>> emails from brocker evere 2/5 minutes.<br>
>> First a mail with "ovirt-hosted-engine state
transition<br>
>> StartState-ReinitializeFSM" and after
"ovirt-hosted-engine state transition<br>
>> ReinitializeFSM-EngineStarting"<br>
>><br>
>><br>
>> Il 17/12/2015 10.51, Stefano Danzi ha scritto:<br>
>>><br>
>>> Hello,<br>
>>> I have one testing host (only one host) with self
hosted engine and 2 VM<br>
>>> (one linux and one windows).<br>
>>><br>
>>> After upgrade ovirt from 3.6.0 to 3.6.1 the network
connection works<br>
>>> discontinuously.<br>
>>> Every 10 minutes HA agent restart hosted engine VM
because result down.<br>
>>> But the machine is UP,<br>
>>> only the network stop to work for some minutes.<br>
>>> I activate global maintenace mode to prevent engine
reboot. If I ssh to<br>
>>> the hosted engine sometimes<br>
>>> the connection work and sometimes no. Using VNC
connection to engine I<br>
>>> see that sometime VM reach external network<br>
>>> and sometimes no.<br>
>>> If I do a tcpdump on phisical ethernet interface I
don't see any packet<br>
>>> when network on vm don't work.<br>
>>><br>
>>> Same thing happens fo others two VM.<br>
>>><br>
>>> Before the upgrade I never had network problems.<br>
>>> _______________________________________________<br>
>>> Users mailing list<br>
>>> <a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> <a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
>>><br>
>><br>
>> _______________________________________________<br>
>> Users mailing list<br>
>> <a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> <a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
<br>
<br>
<br>
-- <br>
Didi<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">
</pre>
</body>
</html>
--------------060609010302050806050207--
3:06 p.m.
On Mon, Dec 28, 2015 at 3:48 PM, Stefano Danzi <s.danzi(a)hawai.it> wrote:
Problem solved!!!
The file hosted-engine.conf had a wrong fqdn.
I don't think that this happened during upgrade... mybe thay my colleague
did something of wrong...
Thanks for the report :-)
Il 20/12/2015 14.52, Stefano Danzi ha scritto:
Network problems was solved after changing Bond mode (and it's strange. I
have to investigate around qemu-kvm, cento 7.2 and switch firmware ), but
broker problem still exist. If I turn on the host, ha agent start engine
vm. When engine VM is up, broker strats to send email. Now I haven't here
detailed logs.
-------- Messaggio originale --------
Da: Yedidyah Bar David <didi(a)redhat.com>
Data: 20/12/2015 11:20 (GMT+01:00)
A: Stefano Danzi <s.danzi(a)hawai.it>, Dan Kenigsberg <danken(a)redhat.com>
Cc: users <users(a)ovirt.org>
Oggetto: Re: [ovirt-users] Network instability after upgrade 3.6.0 -> 3.6.1
On Fri, Dec 18, 2015 at 5:31 PM, Stefano Danzi <s.danzi(a)hawai.it> wrote:
> I found this in vdsm.log and I think that could be the problem:
>
> Thread-3771::ERROR::2015-12-18
>
>
16:18:58,597::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate)
> Connection closed: Connection closed
> Thread-3771::ERROR::2015-12-18 16:18:58,597::API::1847::vds::(_getHaInfo)
> failed to retrieve Hosted Engine HA info
> Traceback (most recent call last):
> File "/usr/share/vdsm/API.py", line 1827, in _getHaInfo
> stats = instance.get_all_stats()
> File
>
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 103, in get_all_stats
> self._configure_broker_conn(broker)
> File
>
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
> line 180, in _configure_broker_conn
> dom_type=dom_type)
> File
>
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 176, in set_storage_domain
> .format(sd_type, options, e))
> RequestError: Failed to set storage domain FilesystemBackend, options
> {'dom_type': 'nfs3', 'sd_uuid':
'46f55a31-f35f-465c-b3e2-df45c05e06a7'}:
> Connection closed
My guess is that this is a consequence of your networking problems.
Adding Dan.
>
>
> Il 17/12/2015 18.51, Stefano Danzi ha scritto:
>>
>> I partially solve the problem.
>>
>> My host machine has 2 network interfaces with a bond. The bond was
>> configured with mode=4 (802.3ad) and switch was configured in the same
>> way.
>> If I remove one network cable the network become stable. With both cables
>> attached the network is instable.
>>
>> I removed the link aggregation configuration from switch and change the
>> bond in mode=2 (balance-xor). Now the network are stable.
>> The strange thing is that previous configuration worked fine for one
>> year... since the last upgrade.
>>
>> Now ha-agent don't reboot the hosted-engine anymore, but I receive two
>> emails from brocker evere 2/5 minutes.
>> First a mail with "ovirt-hosted-engine state transition
>> StartState-ReinitializeFSM" and after "ovirt-hosted-engine state
>> transition
>> ReinitializeFSM-EngineStarting"
>>
>>
>> Il 17/12/2015 10.51, Stefano Danzi ha scritto:
>>>
>>> Hello,
>>> I have one testing host (only one host) with self hosted engine and 2 VM
>>> (one linux and one windows).
>>>
>>> After upgrade ovirt from 3.6.0 to 3.6.1 the network connection works
>>> discontinuously.
>>> Every 10 minutes HA agent restart hosted engine VM because result down.
>>> But the machine is UP,
>>> only the network stop to work for some minutes.
>>> I activate global maintenace mode to prevent engine reboot. If I ssh to
>>> the hosted engine sometimes
>>> the connection work and sometimes no. Using VNC connection to engine I
>>> see that sometime VM reach external network
>>> and sometimes no.
>>> If I do a tcpdump on phisical ethernet interface I don't see any packet
>>> when network on vm don't work.
>>>
>>> Same thing happens fo others two VM.
>>>
>>> Before the upgrade I never had network problems.
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Didi
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Didi
3063
days inactive
3071
days old
3 comments
2 participants
participants (2)
-
Stefano Danzi -
Yedidyah Bar David