10:21 p.m.
----_com.android.email_17262256674260
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64
SSBoYXZlIG9ubHkgb25lIHN3aXRjaCBzbyB0d28gaW50ZXJmYWNlcyBhcmUgY29ubmVjdGVkIHRv
IHRoZSBzYW1lIHN3aXRjaC4gVGhlIGNvbmZpZ3VyYXRpb24gaW4gc3dpdGNoIGlzIGNvcnJlY3Rl
ZC4gwqBJIG9wZW5lZCBhIHRpY2tldCBmb3Igc3dpdGNoIFRlY2ggc3VwcG9ydCBhbmQgdGhlIGNv
bmZpZ3VyYXRpb24gd2FzIHZhbGlkYXRlZC4KVGhpcyBjb25maWd1cmF0aW9uIHdvcmtlZCB3aXRo
b3V0IHByb2JsZW1zIGgyNCBmb3Igb25lIHllYXIhICEhISEgwqBBbGwgcHJvYmxlbXMgc3RhcnRl
ZCBhZnRlciBhIGtlcm5lbCB1cGRhdGUuLi4uIHNvIHNvbWV0aGluZyB3YXMgY2hhbmdlZCBpbiBr
ZXJuZWwuIC4uLi4KCi0tLS0tLS0tIE1lc3NhZ2dpbyBvcmlnaW5hbGUgLS0tLS0tLS0KRGE6IERh
biBLZW5pZ3NiZXJnIDxkYW5rZW5AcmVkaGF0LmNvbT4gCkRhdGE6IDA0LzAyLzIwMTYgIDIyOjAy
ICAoR01UKzAxOjAwKSAKQTogU3RlZmFubyBEYW56aSA8cy5kYW56aUBoYXdhaS5pdD4sIHlkYXJ5
QHJlZGhhdC5jb20gCkNjOiBKb24gQXJjaGVyIDxqb25Acm9zc2x1Zy5vcmcudWs+LCBtYnVybWFu
QHJlZGhhdC5jb20sIHVzZXJzQG92aXJ0Lm9yZyAKT2dnZXR0bzogUmU6IFtvdmlydC11c2Vyc10g
UjogUmU6IE5ldHdvcmsgaW5zdGFiaWxpdHkgYWZ0ZXIgdXBncmFkZSAzLjYuMCAtPgogIDMuNi4x
IAoKT24gVGh1LCBGZWIgMDQsIDIwMTYgYXQgMDY6MjY6MTRQTSArMDEwMCwgU3RlZmFubyBEYW56
aSB3cm90ZToKPiAKPiAKPiBJbCAwNC8wMi8yMDE2IDE2LjU1LCBEYW4gS2VuaWdzYmVyZyBoYSBz
Y3JpdHRvOgo+ID5PbiBXZWQsIEphbiAwNiwgMjAxNiBhdCAwODo0NToxNkFNICswMjAwLCBEYW4g
S2VuaWdzYmVyZyB3cm90ZToKPiA+Pk9uIE1vbiwgSmFuIDA0LCAyMDE2IGF0IDAxOjU0OjM3UE0g
KzAyMDAsIERhbiBLZW5pZ3NiZXJnIHdyb3RlOgo+ID4+Pk9uIE1vbiwgSmFuIDA0LCAyMDE2IGF0
IDEyOjMxOjM4UE0gKzAxMDAsIFN0ZWZhbm8gRGFuemkgd3JvdGU6Cj4gPj4+PkkgZGlkIHNvbWUg
dGVzdHM6Cj4gPj4+Pgo+ID4+Pj5rZXJuZWwtMy4xMC4wLTMyNy4zLjEuZWw3Lng4Nl82NCAtPiBi
b25kIG1vZGUgNCBkb2Vzbid0IHdvcmsgKGlmIEkgZGV0YWNoCj4gPj4+Pm9uZSBuZXR3b3JrIGNh
YmxlIHRoZSBuZXR3b3JrIGlzIHN0YWJsZSkKPiA+Pj4+a2VybmVsLTMuMTAuMC0yMjkuMjAuMS5l
bDcueDg2XzY0IC0+IGJvbmQgbW9kZSA0IHdvcmtzIGZpbmUKPiA+Pj5Xb3VsZCB5b3UgYmUga2lu
ZCB0byBmaWxlIGEga2VybmVsIGJ1ZyBpbiBidWd6aWxsYS5yZWRoYXQuY29tPwo+ID4+PlN1bW1h
cml6ZSB0aGUgaW5mb3JtYXRpb24gZnJvbSB0aGlzIHRocmVhZCAoZS5nLiB5b3VyIGlmY2ZncyBh
bmQgaW4gd2hhdAo+ID4+PndheSBkb2VzIG1vZGUgNCBkb2Vzbid0IHdvcmspLgo+ID4+Pgo+ID4+
PlRvIGdldCB0aGUgYnVnIHNvbHZlZCBxdWlja2x5IHdlJ2QgYmV0dGVyIGZpbmQgcGF5aW5nIFJI
RUw3IGN1c3RvbWVyCj4gPj4+c3Vic2NyaWJpbmcgdG8gaXQuIEJ1dCBJJ2xsIHRyeSB0byBwdXNo
IGZyb20gbXkgZGlyZWN0aW9uLgo+ID4+U3RlZmFubyBoYXMgYmVlbiBraW5kIHRvIG9wZW4KPiA+
Pgo+ID4+wqDCoMKgwqAgQnVnIDEyOTU0MjMgLSBVbnN0YWJsZSBuZXR3b3JrIGxpbmsgdXNpbmcg
Ym9uZCBtb2RlID0gNAo+ID4+wqDCoMKgwqAgaHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29tL3No
b3dfYnVnLmNnaT9pZD0xMjk1NDIzCj4gPj4KPiA+PndoaWNoIHdlIGZhaWwgdG8gcmVwcm9kdWNl
IG9uIG91ciBvd24gbGFiLiBJJ2QgYmUgcGxlYXNlZCBpZiBhbnlib2R5IHdobwo+ID4+ZXhwZXJp
ZW5jZXMgaXQsIGFuZCB0aGVpciBuZXR3b3JraW5nIGNvbmZpZyB0byB0aGUgYnVnIChpZiBpdCBp
cwo+ID4+ZGlmZmVyZW50KS4gQ2FuIHlvdSBhbHNvIGxheSBvdXQgeW91ciBzd2l0Y2gncyBoYXJk
d2FyZSBhbmQKPiA+PmNvbmZpZ3VyYXRpb24/Cj4gPlN0ZWZhbm8sIGNvdWxkIHlvdSBzaGFyZSB5
b3VyIC9wcm9jL25ldC9ib25kaW5nLyogZmlsZXMgd2l0aCB1cz8KPiA+SSBoZWFyZCBhYm91dCBz
aW1pbGFyIHJlcG9ydHMgd2VyZSB0aGUgYm9uZCBzbGF2ZXMgaGFkIG1pc21hdGNoaW5nCj4gPmFn
Z3JlZ2F0b3IgaWQuIENvdWxkIGl0IGJlIHlvdXIgY2FzZSBhcyB3ZWxsPwo+ID4KPiAKPiBIZXJl
Ogo+IAo+IFtyb290QG92aXJ0MDEgfl0jIGNhdCAvcHJvYy9uZXQvYm9uZGluZy9ib25kMAo+IEV0
aGVybmV0IENoYW5uZWwgQm9uZGluZyBEcml2ZXI6IHYzLjcuMSAoQXByaWwgMjcsIDIwMTEpCj4g
Cj4gQm9uZGluZyBNb2RlOiBJRUVFIDgwMi4zYWQgRHluYW1pYyBsaW5rIGFnZ3JlZ2F0aW9uCj4g
VHJhbnNtaXQgSGFzaCBQb2xpY3k6IGxheWVyMiAoMCkKPiBNSUkgU3RhdHVzOiB1cAo+IE1JSSBQ
b2xsaW5nIEludGVydmFsIChtcyk6IDEwMAo+IFVwIERlbGF5IChtcyk6IDAKPiBEb3duIERlbGF5
IChtcyk6IDAKPiAKPiA4MDIuM2FkIGluZm8KPiBMQUNQIHJhdGU6IHNsb3cKPiBNaW4gbGlua3M6
IDAKPiBBZ2dyZWdhdG9yIHNlbGVjdGlvbiBwb2xpY3kgKGFkX3NlbGVjdCk6IHN0YWJsZQo+IEFj
dGl2ZSBBZ2dyZWdhdG9yIEluZm86Cj7CoMKgwqDCoMKgwqDCoMKgIEFnZ3JlZ2F0b3IgSUQ6IDIK
PsKgwqDCoMKgwqDCoMKgwqAgTnVtYmVyIG9mIHBvcnRzOiAxCj7CoMKgwqDCoMKgwqDCoMKgIEFj
dG9yIEtleTogOQo+wqDCoMKgwqDCoMKgwqDCoCBQYXJ0bmVyIEtleTogMQo+wqDCoMKgwqDCoMKg
wqDCoCBQYXJ0bmVyIE1hYyBBZGRyZXNzOiAwMDowMDowMDowMDowMDowMAo+IAo+IFNsYXZlIElu
dGVyZmFjZTogZW5wNHMwCj4gTUlJIFN0YXR1czogdXAKPiBTcGVlZDogMTAwMCBNYnBzCj4gRHVw
bGV4OiBmdWxsCj4gTGluayBGYWlsdXJlIENvdW50OiAyCj4gUGVybWFuZW50IEhXIGFkZHI6ICoq
OioqOioqOioqOioqOmYxCj4gU2xhdmUgcXVldWUgSUQ6IDAKPiBBZ2dyZWdhdG9yIElEOiAxCgot
LS0tLS0tLS0tLS0tLS1eXl4KCgo+IEFjdG9yIENodXJuIFN0YXRlOiBjaHVybmVkCj4gUGFydG5l
ciBDaHVybiBTdGF0ZTogY2h1cm5lZAo+IEFjdG9yIENodXJuZWQgQ291bnQ6IDQKPiBQYXJ0bmVy
IENodXJuZWQgQ291bnQ6IDUKPiBkZXRhaWxzIGFjdG9yIGxhY3AgcGR1Ogo+wqDCoMKgwqAgc3lz
dGVtIHByaW9yaXR5OiA2NTUzNQo+wqDCoMKgwqAgcG9ydCBrZXk6IDkKPsKgwqDCoMKgIHBvcnQg
cHJpb3JpdHk6IDI1NQo+wqDCoMKgwqAgcG9ydCBudW1iZXI6IDEKPsKgwqDCoMKgIHBvcnQgc3Rh
dGU6IDY5Cj4gZGV0YWlscyBwYXJ0bmVyIGxhY3AgcGR1Ogo+wqDCoMKgwqAgc3lzdGVtIHByaW9y
aXR5OiA2NTUzNQo+wqDCoMKgwqAgb3BlciBrZXk6IDEKPsKgwqDCoMKgIHBvcnQgcHJpb3JpdHk6
IDI1NQo+wqDCoMKgwqAgcG9ydCBudW1iZXI6IDEKPsKgwqDCoMKgIHBvcnQgc3RhdGU6IDEKPiAK
PiBTbGF2ZSBJbnRlcmZhY2U6IGVucDVzMAo+IE1JSSBTdGF0dXM6IHVwCj4gU3BlZWQ6IDEwMDAg
TWJwcwo+IER1cGxleDogZnVsbAo+IExpbmsgRmFpbHVyZSBDb3VudDogMQo+IFBlcm1hbmVudCBI
VyBhZGRyOiAqKjoqKjoqKjoqKjoqKjpmMgo+IFNsYXZlIHF1ZXVlIElEOiAwCj4gQWdncmVnYXRv
ciBJRDogMgoKLS0tLS0tLS0tLS0tLS0tXl5eCgoKaXQgc291bmRzIGF3ZnVsbHkgZmFtaWxpYXIg
LSBtaXNtYXRjaGluZyBhZ2dyZWdhdG9yIElEcywgYW5kIGFuIGFsbC16ZXJvCnBhcnRuZXIgbWFj
LiBDYW4geW91IGRvdWJsZS1jaGVjayB0aGF0IGJvdGggeW91ciBuaWNzIGFyZSB3aXJlZCB0byB0
aGUKc2FtZSBzd2l0Y2gsIHdoaWNoIGlzIHByb3Blcmx5IGNvbmZpZ3VyZWQgdG8gdXNlIGxhY3Ag
b24gdGhlc2UgdHdvCnBvcnRzPwoK
----_com.android.email_17262256674260
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0
L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPjwvaGVhZD48Ym9keT5JIGhhdmUgb25seSBvbmUgc3dpdGNo
IHNvIHR3byBpbnRlcmZhY2VzIGFyZSBjb25uZWN0ZWQgdG8gdGhlIHNhbWUgc3dpdGNoLiBUaGUg
Y29uZmlndXJhdGlvbiBpbiBzd2l0Y2ggaXMgY29ycmVjdGVkLiAmbmJzcDtJIG9wZW5lZCBhIHRp
Y2tldCBmb3Igc3dpdGNoIFRlY2ggc3VwcG9ydCBhbmQgdGhlIGNvbmZpZ3VyYXRpb24gd2FzIHZh
bGlkYXRlZC48ZGl2Pjxicj48L2Rpdj48ZGl2PlRoaXMgY29uZmlndXJhdGlvbiB3b3JrZWQgd2l0
aG91dCBwcm9ibGVtcyBoMjQgZm9yIG9uZSB5ZWFyISAhISEhICZuYnNwO0FsbCBwcm9ibGVtcyBz
dGFydGVkIGFmdGVyIGEga2VybmVsIHVwZGF0ZS4uLi4gc28gc29tZXRoaW5nIHdhcyBjaGFuZ2Vk
IGluIGtlcm5lbC4gLi4uLjwvZGl2Pjxicj48YnI+PGRpdiBzdHlsZT0iZm9udC1zaXplOjEwMCU7
dGV4dC1hbGlnbjpsZWZ0O2NvbG9yOiMwMDAwMDAiPi0tLS0tLS0tIE1lc3NhZ2dpbyBvcmlnaW5h
bGUgLS0tLS0tLS08YnI+RGE6IERhbiBLZW5pZ3NiZXJnICZsdDtkYW5rZW5AcmVkaGF0LmNvbSZn
dDsgPGJyPkRhdGE6IDA0LzAyLzIwMTYgIDIyOjAyICAoR01UKzAxOjAwKSA8YnI+QTogU3RlZmFu
byBEYW56aSAmbHQ7cy5kYW56aUBoYXdhaS5pdCZndDssIHlkYXJ5QHJlZGhhdC5jb20gPGJyPkNj
OiBKb24gQXJjaGVyICZsdDtqb25Acm9zc2x1Zy5vcmcudWsmZ3Q7LCBtYnVybWFuQHJlZGhhdC5j
b20sIHVzZXJzQG92aXJ0Lm9yZyA8YnI+T2dnZXR0bzogUmU6IFtvdmlydC11c2Vyc10gUjogUmU6
IE5ldHdvcmsgaW5zdGFiaWxpdHkgYWZ0ZXIgdXBncmFkZSAzLjYuMCAtJmd0OwogIDMuNi4xIDxi
cj48YnI+PC9kaXY+T24gVGh1LCBGZWIgMDQsIDIwMTYgYXQgMDY6MjY6MTRQTSArMDEwMCwgU3Rl
ZmFubyBEYW56aSB3cm90ZTo8YnI+Jmd0OyA8YnI+Jmd0OyA8YnI+Jmd0OyBJbCAwNC8wMi8yMDE2
IDE2LjU1LCBEYW4gS2VuaWdzYmVyZyBoYSBzY3JpdHRvOjxicj4mZ3Q7ICZndDtPbiBXZWQsIEph
biAwNiwgMjAxNiBhdCAwODo0NToxNkFNICswMjAwLCBEYW4gS2VuaWdzYmVyZyB3cm90ZTo8YnI+
Jmd0OyAmZ3Q7Jmd0O09uIE1vbiwgSmFuIDA0LCAyMDE2IGF0IDAxOjU0OjM3UE0gKzAyMDAsIERh
biBLZW5pZ3NiZXJnIHdyb3RlOjxicj4mZ3Q7ICZndDsmZ3Q7Jmd0O09uIE1vbiwgSmFuIDA0LCAy
MDE2IGF0IDEyOjMxOjM4UE0gKzAxMDAsIFN0ZWZhbm8gRGFuemkgd3JvdGU6PGJyPiZndDsgJmd0
OyZndDsmZ3Q7Jmd0O0kgZGlkIHNvbWUgdGVzdHM6PGJyPiZndDsgJmd0OyZndDsmZ3Q7Jmd0Ozxi
cj4mZ3Q7ICZndDsmZ3Q7Jmd0OyZndDtrZXJuZWwtMy4xMC4wLTMyNy4zLjEuZWw3Lng4Nl82NCAt
Jmd0OyBib25kIG1vZGUgNCBkb2Vzbid0IHdvcmsgKGlmIEkgZGV0YWNoPGJyPiZndDsgJmd0OyZn
dDsmZ3Q7Jmd0O29uZSBuZXR3b3JrIGNhYmxlIHRoZSBuZXR3b3JrIGlzIHN0YWJsZSk8YnI+Jmd0
OyAmZ3Q7Jmd0OyZndDsmZ3Q7a2VybmVsLTMuMTAuMC0yMjkuMjAuMS5lbDcueDg2XzY0IC0mZ3Q7
IGJvbmQgbW9kZSA0IHdvcmtzIGZpbmU8YnI+Jmd0OyAmZ3Q7Jmd0OyZndDtXb3VsZCB5b3UgYmUg
a2luZCB0byBmaWxlIGEga2VybmVsIGJ1ZyBpbiBidWd6aWxsYS5yZWRoYXQuY29tPzxicj4mZ3Q7
ICZndDsmZ3Q7Jmd0O1N1bW1hcml6ZSB0aGUgaW5mb3JtYXRpb24gZnJvbSB0aGlzIHRocmVhZCAo
ZS5nLiB5b3VyIGlmY2ZncyBhbmQgaW4gd2hhdDxicj4mZ3Q7ICZndDsmZ3Q7Jmd0O3dheSBkb2Vz
IG1vZGUgNCBkb2Vzbid0IHdvcmspLjxicj4mZ3Q7ICZndDsmZ3Q7Jmd0Ozxicj4mZ3Q7ICZndDsm
Z3Q7Jmd0O1RvIGdldCB0aGUgYnVnIHNvbHZlZCBxdWlja2x5IHdlJ2QgYmV0dGVyIGZpbmQgcGF5
aW5nIFJIRUw3IGN1c3RvbWVyPGJyPiZndDsgJmd0OyZndDsmZ3Q7c3Vic2NyaWJpbmcgdG8gaXQu
IEJ1dCBJJ2xsIHRyeSB0byBwdXNoIGZyb20gbXkgZGlyZWN0aW9uLjxicj4mZ3Q7ICZndDsmZ3Q7
U3RlZmFubyBoYXMgYmVlbiBraW5kIHRvIG9wZW48YnI+Jmd0OyAmZ3Q7Jmd0Ozxicj4mZ3Q7ICZn
dDsmZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IEJ1ZyAxMjk1NDIzIC0gVW5zdGFibGUgbmV0
d29yayBsaW5rIHVzaW5nIGJvbmQgbW9kZSA9IDQ8YnI+Jmd0OyAmZ3Q7Jmd0OyZuYnNwOyZuYnNw
OyZuYnNwOyZuYnNwOyBodHRwczovL2J1Z3ppbGxhLnJlZGhhdC5jb20vc2hvd19idWcuY2dpP2lk
PTEyOTU0MjM8YnI+Jmd0OyAmZ3Q7Jmd0Ozxicj4mZ3Q7ICZndDsmZ3Q7d2hpY2ggd2UgZmFpbCB0
byByZXByb2R1Y2Ugb24gb3VyIG93biBsYWIuIEknZCBiZSBwbGVhc2VkIGlmIGFueWJvZHkgd2hv
PGJyPiZndDsgJmd0OyZndDtleHBlcmllbmNlcyBpdCwgYW5kIHRoZWlyIG5ldHdvcmtpbmcgY29u
ZmlnIHRvIHRoZSBidWcgKGlmIGl0IGlzPGJyPiZndDsgJmd0OyZndDtkaWZmZXJlbnQpLiBDYW4g
eW91IGFsc28gbGF5IG91dCB5b3VyIHN3aXRjaCdzIGhhcmR3YXJlIGFuZDxicj4mZ3Q7ICZndDsm
Z3Q7Y29uZmlndXJhdGlvbj88YnI+Jmd0OyAmZ3Q7U3RlZmFubywgY291bGQgeW91IHNoYXJlIHlv
dXIgL3Byb2MvbmV0L2JvbmRpbmcvKiBmaWxlcyB3aXRoIHVzPzxicj4mZ3Q7ICZndDtJIGhlYXJk
IGFib3V0IHNpbWlsYXIgcmVwb3J0cyB3ZXJlIHRoZSBib25kIHNsYXZlcyBoYWQgbWlzbWF0Y2hp
bmc8YnI+Jmd0OyAmZ3Q7YWdncmVnYXRvciBpZC4gQ291bGQgaXQgYmUgeW91ciBjYXNlIGFzIHdl
bGw/PGJyPiZndDsgJmd0Ozxicj4mZ3Q7IDxicj4mZ3Q7IEhlcmU6PGJyPiZndDsgPGJyPiZndDsg
W3Jvb3RAb3ZpcnQwMSB+XSMgY2F0IC9wcm9jL25ldC9ib25kaW5nL2JvbmQwPGJyPiZndDsgRXRo
ZXJuZXQgQ2hhbm5lbCBCb25kaW5nIERyaXZlcjogdjMuNy4xIChBcHJpbCAyNywgMjAxMSk8YnI+
Jmd0OyA8YnI+Jmd0OyBCb25kaW5nIE1vZGU6IElFRUUgODAyLjNhZCBEeW5hbWljIGxpbmsgYWdn
cmVnYXRpb248YnI+Jmd0OyBUcmFuc21pdCBIYXNoIFBvbGljeTogbGF5ZXIyICgwKTxicj4mZ3Q7
IE1JSSBTdGF0dXM6IHVwPGJyPiZndDsgTUlJIFBvbGxpbmcgSW50ZXJ2YWwgKG1zKTogMTAwPGJy
PiZndDsgVXAgRGVsYXkgKG1zKTogMDxicj4mZ3Q7IERvd24gRGVsYXkgKG1zKTogMDxicj4mZ3Q7
IDxicj4mZ3Q7IDgwMi4zYWQgaW5mbzxicj4mZ3Q7IExBQ1AgcmF0ZTogc2xvdzxicj4mZ3Q7IE1p
biBsaW5rczogMDxicj4mZ3Q7IEFnZ3JlZ2F0b3Igc2VsZWN0aW9uIHBvbGljeSAoYWRfc2VsZWN0
KTogc3RhYmxlPGJyPiZndDsgQWN0aXZlIEFnZ3JlZ2F0b3IgSW5mbzo8YnI+Jmd0OyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBBZ2dyZWdhdG9yIElEOiAy
PGJyPiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsg
TnVtYmVyIG9mIHBvcnRzOiAxPGJyPiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsgQWN0b3IgS2V5OiA5PGJyPiZndDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgUGFydG5lciBLZXk6IDE8YnI+Jmd0OyZuYnNw
OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBQYXJ0bmVyIE1hYyBB
ZGRyZXNzOiAwMDowMDowMDowMDowMDowMDxicj4mZ3Q7IDxicj4mZ3Q7IFNsYXZlIEludGVyZmFj
ZTogZW5wNHMwPGJyPiZndDsgTUlJIFN0YXR1czogdXA8YnI+Jmd0OyBTcGVlZDogMTAwMCBNYnBz
PGJyPiZndDsgRHVwbGV4OiBmdWxsPGJyPiZndDsgTGluayBGYWlsdXJlIENvdW50OiAyPGJyPiZn
dDsgUGVybWFuZW50IEhXIGFkZHI6ICoqOioqOioqOioqOioqOmYxPGJyPiZndDsgU2xhdmUgcXVl
dWUgSUQ6IDA8YnI+Jmd0OyBBZ2dyZWdhdG9yIElEOiAxPGJyPjxicj4tLS0tLS0tLS0tLS0tLS1e
Xl48YnI+PGJyPjxicj4mZ3Q7IEFjdG9yIENodXJuIFN0YXRlOiBjaHVybmVkPGJyPiZndDsgUGFy
dG5lciBDaHVybiBTdGF0ZTogY2h1cm5lZDxicj4mZ3Q7IEFjdG9yIENodXJuZWQgQ291bnQ6IDQ8
YnI+Jmd0OyBQYXJ0bmVyIENodXJuZWQgQ291bnQ6IDU8YnI+Jmd0OyBkZXRhaWxzIGFjdG9yIGxh
Y3AgcGR1Ojxicj4mZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHN5c3RlbSBwcmlvcml0eTog
NjU1MzU8YnI+Jmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBwb3J0IGtleTogOTxicj4mZ3Q7
Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHBvcnQgcHJpb3JpdHk6IDI1NTxicj4mZ3Q7Jm5ic3A7
Jm5ic3A7Jm5ic3A7Jm5ic3A7IHBvcnQgbnVtYmVyOiAxPGJyPiZndDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsgcG9ydCBzdGF0ZTogNjk8YnI+Jmd0OyBkZXRhaWxzIHBhcnRuZXIgbGFjcCBwZHU6
PGJyPiZndDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgc3lzdGVtIHByaW9yaXR5OiA2NTUzNTxi
cj4mZ3Q7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IG9wZXIga2V5OiAxPGJyPiZndDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsgcG9ydCBwcmlvcml0eTogMjU1PGJyPiZndDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsgcG9ydCBudW1iZXI6IDE8YnI+Jmd0OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw
OyBwb3J0IHN0YXRlOiAxPGJyPiZndDsgPGJyPiZndDsgU2xhdmUgSW50ZXJmYWNlOiBlbnA1czA8
YnI+Jmd0OyBNSUkgU3RhdHVzOiB1cDxicj4mZ3Q7IFNwZWVkOiAxMDAwIE1icHM8YnI+Jmd0OyBE
dXBsZXg6IGZ1bGw8YnI+Jmd0OyBMaW5rIEZhaWx1cmUgQ291bnQ6IDE8YnI+Jmd0OyBQZXJtYW5l
bnQgSFcgYWRkcjogKio6Kio6Kio6Kio6Kio6ZjI8YnI+Jmd0OyBTbGF2ZSBxdWV1ZSBJRDogMDxi
cj4mZ3Q7IEFnZ3JlZ2F0b3IgSUQ6IDI8YnI+PGJyPi0tLS0tLS0tLS0tLS0tLV5eXjxicj48YnI+
PGJyPml0IHNvdW5kcyBhd2Z1bGx5IGZhbWlsaWFyIC0gbWlzbWF0Y2hpbmcgYWdncmVnYXRvciBJ
RHMsIGFuZCBhbiBhbGwtemVybzxicj5wYXJ0bmVyIG1hYy4gQ2FuIHlvdSBkb3VibGUtY2hlY2sg
dGhhdCBib3RoIHlvdXIgbmljcyBhcmUgd2lyZWQgdG8gdGhlPGJyPnNhbWUgc3dpdGNoLCB3aGlj
aCBpcyBwcm9wZXJseSBjb25maWd1cmVkIHRvIHVzZSBsYWNwIG9uIHRoZXNlIHR3bzxicj5wb3J0
cz88YnI+PGJyPjwvYm9keT48L2h0bWw+
----_com.android.email_17262256674260--
2:40 p.m.
New subject: R: Re: R: Re: Network instability after upgrade 3.6.0 ->? 3.6.1
On Thu, Feb 04, 2016 at 10:21:10PM +0100, Stefano Danzi wrote:
I have only one switch so two interfaces are connected to the same
switch. The configuration in switch is corrected. I opened a ticket for switch Tech
support and the configuration was validated.
This configuration worked without problems h24 for one year! !!!! All problems started
after a kernel update.... so something was changed in kernel. ....
Jarod, do you have a clue why AggregatorIDs may be mismatching with
recent el7.2 kernels?
-------- Messaggio originale --------
Da: Dan Kenigsberg <danken(a)redhat.com>
Data: 04/02/2016 22:02 (GMT+01:00)
A: Stefano Danzi <s.danzi(a)hawai.it>, ydary(a)redhat.com
Cc: Jon Archer <jon(a)rosslug.org.uk>, mburman(a)redhat.com, users(a)ovirt.org
Oggetto: Re: [ovirt-users] R: Re: Network instability after upgrade 3.6.0 ->
3.6.1
On Thu, Feb 04, 2016 at 06:26:14PM +0100, Stefano Danzi wrote:
>
>
> Il 04/02/2016 16.55, Dan Kenigsberg ha scritto:
> >On Wed, Jan 06, 2016 at 08:45:16AM +0200, Dan Kenigsberg wrote:
> >>On Mon, Jan 04, 2016 at 01:54:37PM +0200, Dan Kenigsberg wrote:
> >>>On Mon, Jan 04, 2016 at 12:31:38PM +0100, Stefano Danzi wrote:
> >>>>I did some tests:
> >>>>
> >>>>kernel-3.10.0-327.3.1.el7.x86_64 -> bond mode 4 doesn't work
(if I detach
> >>>>one network cable the network is stable)
> >>>>kernel-3.10.0-229.20.1.el7.x86_64 -> bond mode 4 works fine
> >>>Would you be kind to file a kernel bug in bugzilla.redhat.com?
> >>>Summarize the information from this thread (e.g. your ifcfgs and in
what
> >>>way does mode 4 doesn't work).
> >>>
> >>>To get the bug solved quickly we'd better find paying RHEL7
customer
> >>>subscribing to it. But I'll try to push from my direction.
> >>Stefano has been kind to open
> >>
> >> Bug 1295423 - Unstable network link using bond mode = 4
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1295423
> >>
> >>which we fail to reproduce on our own lab. I'd be pleased if anybody
who
> >>experiences it, and their networking config to the bug (if it is
> >>different). Can you also lay out your switch's hardware and
> >>configuration?
> >Stefano, could you share your /proc/net/bonding/* files with us?
> >I heard about similar reports were the bond slaves had mismatching
> >aggregator id. Could it be your case as well?
> >
>
> Here:
>
> [root@ovirt01 ~]# cat /proc/net/bonding/bond0
> Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
>
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> Transmit Hash Policy: layer2 (0)
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 0
> Down Delay (ms): 0
>
> 802.3ad info
> LACP rate: slow
> Min links: 0
> Aggregator selection policy (ad_select): stable
> Active Aggregator Info:
> Aggregator ID: 2
> Number of ports: 1
> Actor Key: 9
> Partner Key: 1
> Partner Mac Address: 00:00:00:00:00:00
>
> Slave Interface: enp4s0
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 2
> Permanent HW addr: **:**:**:**:**:f1
> Slave queue ID: 0
> Aggregator ID: 1
---------------^^^
> Actor Churn State: churned
> Partner Churn State: churned
> Actor Churned Count: 4
> Partner Churned Count: 5
> details actor lacp pdu:
> system priority: 65535
> port key: 9
> port priority: 255
> port number: 1
> port state: 69
> details partner lacp pdu:
> system priority: 65535
> oper key: 1
> port priority: 255
> port number: 1
> port state: 1
>
> Slave Interface: enp5s0
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 1
> Permanent HW addr: **:**:**:**:**:f2
> Slave queue ID: 0
> Aggregator ID: 2
---------------^^^
it sounds awfully familiar - mismatching aggregator IDs, and an all-zero
partner mac. Can you double-check that both your nics are wired to the
same switch, which is properly configured to use lacp on these two
ports?
7:32 p.m.
New subject: R: Re: R: Re: Network instability after upgrade 3.6.0 ->? 3.6.1
On Mon, Feb 08, 2016 at 03:40:15PM +0200, Dan Kenigsberg wrote:
On Thu, Feb 04, 2016 at 10:21:10PM +0100, Stefano Danzi wrote:
> I have only one switch so two interfaces are connected to the same switch. The
configuration in switch is corrected. I opened a ticket for switch Tech support and the
configuration was validated.
> This configuration worked without problems h24 for one year! !!!! All problems
started after a kernel update.... so something was changed in kernel. ....
Jarod, do you have a clue why AggregatorIDs may be mismatching with
recent el7.2 kernels?
Ah, the mail chain includes the reporter of bug 1295423. In that bug,
we've already discovered that even with the older supposedly working
kernel, the two interfaces had different LAG IDs, so I don't think it's
recent kernels causing the IDs to mis-match, but something recent
certainly makes that more fatal than it did before. I'm not sure what the
core problem is here, don't want to chase anything without being able to
say for sure that things are 100% correct on the switch side.
> -------- Messaggio originale --------
> Da: Dan Kenigsberg <danken(a)redhat.com>
> Data: 04/02/2016 22:02 (GMT+01:00)
> A: Stefano Danzi <s.danzi(a)hawai.it>, ydary(a)redhat.com
> Cc: Jon Archer <jon(a)rosslug.org.uk>, mburman(a)redhat.com, users(a)ovirt.org
> Oggetto: Re: [ovirt-users] R: Re: Network instability after upgrade 3.6.0 ->
> 3.6.1
>
> On Thu, Feb 04, 2016 at 06:26:14PM +0100, Stefano Danzi wrote:
> >
> >
> > Il 04/02/2016 16.55, Dan Kenigsberg ha scritto:
> > >On Wed, Jan 06, 2016 at 08:45:16AM +0200, Dan Kenigsberg wrote:
> > >>On Mon, Jan 04, 2016 at 01:54:37PM +0200, Dan Kenigsberg wrote:
> > >>>On Mon, Jan 04, 2016 at 12:31:38PM +0100, Stefano Danzi wrote:
> > >>>>I did some tests:
> > >>>>
> > >>>>kernel-3.10.0-327.3.1.el7.x86_64 -> bond mode 4 doesn't
work (if I detach
> > >>>>one network cable the network is stable)
> > >>>>kernel-3.10.0-229.20.1.el7.x86_64 -> bond mode 4 works fine
> > >>>Would you be kind to file a kernel bug in bugzilla.redhat.com?
> > >>>Summarize the information from this thread (e.g. your ifcfgs and in
what
> > >>>way does mode 4 doesn't work).
> > >>>
> > >>>To get the bug solved quickly we'd better find paying RHEL7
customer
> > >>>subscribing to it. But I'll try to push from my direction.
> > >>Stefano has been kind to open
> > >>
> > >> Bug 1295423 - Unstable network link using bond mode = 4
> > >> https://bugzilla.redhat.com/show_bug.cgi?id=1295423
> > >>
> > >>which we fail to reproduce on our own lab. I'd be pleased if
anybody who
> > >>experiences it, and their networking config to the bug (if it is
> > >>different). Can you also lay out your switch's hardware and
> > >>configuration?
> > >Stefano, could you share your /proc/net/bonding/* files with us?
> > >I heard about similar reports were the bond slaves had mismatching
> > >aggregator id. Could it be your case as well?
> > >
> >
> > Here:
> >
> > [root@ovirt01 ~]# cat /proc/net/bonding/bond0
> > Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
> >
> > Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> > Transmit Hash Policy: layer2 (0)
> > MII Status: up
> > MII Polling Interval (ms): 100
> > Up Delay (ms): 0
> > Down Delay (ms): 0
> >
> > 802.3ad info
> > LACP rate: slow
> > Min links: 0
> > Aggregator selection policy (ad_select): stable
> > Active Aggregator Info:
> > Aggregator ID: 2
> > Number of ports: 1
> > Actor Key: 9
> > Partner Key: 1
> > Partner Mac Address: 00:00:00:00:00:00
> >
> > Slave Interface: enp4s0
> > MII Status: up
> > Speed: 1000 Mbps
> > Duplex: full
> > Link Failure Count: 2
> > Permanent HW addr: **:**:**:**:**:f1
> > Slave queue ID: 0
> > Aggregator ID: 1
>
> ---------------^^^
>
>
> > Actor Churn State: churned
> > Partner Churn State: churned
> > Actor Churned Count: 4
> > Partner Churned Count: 5
> > details actor lacp pdu:
> > system priority: 65535
> > port key: 9
> > port priority: 255
> > port number: 1
> > port state: 69
> > details partner lacp pdu:
> > system priority: 65535
> > oper key: 1
> > port priority: 255
> > port number: 1
> > port state: 1
> >
> > Slave Interface: enp5s0
> > MII Status: up
> > Speed: 1000 Mbps
> > Duplex: full
> > Link Failure Count: 1
> > Permanent HW addr: **:**:**:**:**:f2
> > Slave queue ID: 0
> > Aggregator ID: 2
>
> ---------------^^^
>
>
> it sounds awfully familiar - mismatching aggregator IDs, and an all-zero
> partner mac. Can you double-check that both your nics are wired to the
> same switch, which is properly configured to use lacp on these two
> ports?
>
--
Jarod Wilson
jarod(a)redhat.com
3019
days inactive
3024
days old
2 comments
3 participants
participants (3)
-
Dan Kenigsberg -
Jarod Wilson -
Stefano Danzi