vm pauses with "vm has paused due to unknown storage error
--_000_CO2PR0801MB0743180FC142ED11D7D5D3E0A62E0CO2PR0801MB0743_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGVsbG8sDQoNCkhhdmUgMyBub2RlcyBydW5uaW5nIGJvdGggb1ZpcnQgYW5kIEdsdXN0ZXIgb24g NCBTU0TigJlzIGVhY2guIEF0IHRoZSBtb21lbnQsIHRoZXJlIGFyZSB0d28gcGh5c2ljYWwgbmlj cywgb25lIGhhcyBwdWJsaWMgaW50ZXJuZXQgYWNjZXNzIGFuZCB0aGUgb3RoZXIgaXMgYSBub24t cm91dGFibGUgbmV0d29yayB1c2VkIGZvciBvdmlydG1nbXQgJiBnbHVzdGVyLg0KDQpJbiB0aGUg bG9naWNhbCBuZXR3b3JrcywgSSBoYXZlIHNlbGVjdGVkIGdsdXN0ZXIgZm9yIHRoZSBub25yb3V0 YWJsZSBuZXR3b3JrIHJ1bm5pbmcgb3ZpcnRtZ210IGFuZCBnbHVzdGVyIGhvd2V2ZXIsIHR3byBW TeKAmXMgcmFuZG9tbHkgcGF1c2UgZm9yIHdoYXQgc2VlbXMgbGlrZSBubyByZWFzb24uIFRoZXkg Y2FuIGJvdGggYmUgcmVzdW1lZCB3aXRob3V0IGlzc3VlLg0KDQpPbmUgdGVzdCBWTSBoYXMgNEdC IG9mIG1lbW9yeSBhbmQgYSBzbWFsbCBkaXNrIOKAkyBubyBwcm9ibGVtcyB3aXRoIHRoaXMgb25l LiBUd28gb3RoZXJzIGhhdmUgODAwR0IgZGlza3MgYW5kIDMyR0Igb2YgUkFNIOKAkyBib3RoIHZt 4oCZcyBleGhpYml0IHRoZSBzYW1lIGlzc3VlLg0KDQpJIGFsc28gc2VlIHRoZXNlIGluIHRoZSBv VmlydCBkYXNoYm9hcmQ6DQoNCg0KRmFpbGVkIHRvIHVwZGF0ZSBPVkYgZGlza3MgOWU2MDMyOGQt MjlhZi00NTMzLTg0ZjktNjMzZDg3ZjU0OGE3LCBPVkYgZGF0YSBpc24ndCB1cGRhdGVkIG9uIHRo b3NlIE9WRiBzdG9yZXMgKERhdGEgQ2VudGVyIHh4eHh4LCBTdG9yYWdlIERvbWFpbiBzci12b2x1 bWUwMSkuDQoNCkp1biAyMywgMjAxNiA5OjU0OjAzIFBNDQoNClZEU00gY29tbWFuZCBmYWlsZWQ6 IENvdWxkIG5vdCBhY3F1aXJlIHJlc291cmNlLiBQcm9iYWJseSByZXNvdXJjZSBmYWN0b3J5IHRo cmV3IGFuIGV4Y2VwdGlvbi46ICgpDQoNCi8vLy8vLy8vLy8vLy8vLw0KDQpWTSB4eHh4eCBoYXMg YmVlbiBwYXVzZWQgZHVlIHRvIHVua25vd24gc3RvcmFnZSBlcnJvci4NCg0KLy8vLy8vLy8vLy8v Ly8vDQoNCkluIHRoZSBlcnJvciBsb2cgb24gdGhlIGVuZ2luZSwgSSBzZWUgdGhlc2U6DQoNCkVS Uk9SIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJva2VyLmF1ZGl0bG9naGFuZGxpbmcu QXVkaXRMb2dEaXJlY3Rvcl0gKEZvcmtKb2luUG9vbC0xLXdvcmtlci03KSBbMTBjYWY5M2VdIENv cnJlbGF0aW9uIElEOiBudWxsLCBDYWxsIFN0YWNrOiBudWxsLCBDdXN0b20gRXZlbnQgSUQ6IC0x LCBNZXNzYWdlOiBWTSB4eHh4eHggaGFzIGJlZW4gcGF1c2VkIGR1ZSB0byB1bmtub3duIHN0b3Jh Z2UgZXJyb3IuDQoNCklORk8gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJva2VyLmF1 ZGl0bG9naGFuZGxpbmcuQXVkaXRMb2dEaXJlY3Rvcl0gKEZvcmtKb2luUG9vbC0xLXdvcmtlci0x MSkgWzEwY2FmOTNlXSBDb3JyZWxhdGlvbiBJRDogbnVsbCwgQ2FsbCBTdGFjazogbnVsbCwgQ3Vz dG9tIEV2ZW50IElEOiAtMSwgTWVzc2FnZTogVk0geHh4eHh4IGhhcyByZWNvdmVyZWQgZnJvbSBw YXVzZWQgYmFjayB0byB1cC4NCg0KLy8vLy8vLy8vLy8vLy8vDQoNCkhvc3RuYW1lcyBhcmUgYWxs IGxvY2FsIHRvIC9ldGMvaG9zdHMgb24gYWxsIHNlcnZlcnMg4oCTIHRoZXkgYWxzbyByZXNvbHZl IHdpdGhvdXQgaXNzdWUgZnJvbSBlYWNoIGhvc3QuDQoNCi8vLy8vLy8vLy8vLy8vDQoNCjIwMTYt MDYtMjMgMjI6MDg6NTksNjExIFdBUk4gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2Vy LmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9yWG1sUnBjXSAoRGVmYXVsdFF1YXJ0 elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBub3QgYXNzb2NpYXRlIGJyaWNr ICdvdmlydDM0MzU6L21udC9kYXRhL3NyLXZvbHVtZTAxJyBvZiB2b2x1bWUgJzkzZTM2Y2RjLWFi MWItNDFlYy1hYzdmLTk2NmNmMzg1NmI1OScgd2l0aCBjb3JyZWN0IG5ldHdvcmsgYXMgbm8gZ2x1 c3RlciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5OS05Y2QwLWI2 M2U5MTliOWFjYScNCjIwMTYtMDYtMjMgMjI6MDg6NTksNjE0IFdBUk4gIFtvcmcub3ZpcnQuZW5n aW5lLmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9yWG1s UnBjXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBu b3QgYXNzb2NpYXRlIGJyaWNrICdvdmlydDM2Mzc6L21udC9kYXRhL3NyLXZvbHVtZTAxJyBvZiB2 b2x1bWUgJzkzZTM2Y2RjLWFiMWItNDFlYy1hYzdmLTk2NmNmMzg1NmI1OScgd2l0aCBjb3JyZWN0 IG5ldHdvcmsgYXMgbm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRl LTA0YjItNGE5OS05Y2QwLWI2M2U5MTliOWFjYScNCjIwMTYtMDYtMjMgMjI6MDg6NTksNjE2IFdB Uk4gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVt ZXNMaXN0UmV0dXJuRm9yWG1sUnBjXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYp IFsxYzFjZjRmXSBDb3VsZCBub3QgYXNzb2NpYXRlIGJyaWNrICdvdmlydDM4Mzk6L21udC9kYXRh L3NyLXZvbHVtZTAxJyBvZiB2b2x1bWUgJzkzZTM2Y2RjLWFiMWItNDFlYy1hYzdmLTk2NmNmMzg1 NmI1OScgd2l0aCBjb3JyZWN0IG5ldHdvcmsgYXMgbm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGlu IGNsdXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5OS05Y2QwLWI2M2U5MTliOWFjYScNCjIwMTYtMDYt MjMgMjI6MDg6NTksNjE4IFdBUk4gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLmds dXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9yWG1sUnBjXSAoRGVmYXVsdFF1YXJ0elNj aGVkdWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBub3QgYXNzb2NpYXRlIGJyaWNrICdv dmlydDM0MzU6L21udC9kYXRhL2Rpc3RyaWJ1dGVkJyBvZiB2b2x1bWUgJ2I4ODdiMDVlLTJlYTYt NDk2ZS05NTUyLTE1NWQ2NThlZWFhNicgd2l0aCBjb3JyZWN0IG5ldHdvcmsgYXMgbm8gZ2x1c3Rl ciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5OS05Y2QwLWI2M2U5 MTliOWFjYScNCjIwMTYtMDYtMjMgMjI6MDg6NTksNjIwIFdBUk4gIFtvcmcub3ZpcnQuZW5naW5l LmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9yWG1sUnBj XSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBub3Qg YXNzb2NpYXRlIGJyaWNrICdvdmlydDM2Mzc6L21udC9kYXRhL2Rpc3RyaWJ1dGVkJyBvZiB2b2x1 bWUgJ2I4ODdiMDVlLTJlYTYtNDk2ZS05NTUyLTE1NWQ2NThlZWFhNicgd2l0aCBjb3JyZWN0IG5l dHdvcmsgYXMgbm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRlLTA0 YjItNGE5OS05Y2QwLWI2M2U5MTliOWFjYScNCjIwMTYtMDYtMjMgMjI6MDg6NTksNjIyIFdBUk4g IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNM aXN0UmV0dXJuRm9yWG1sUnBjXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsx YzFjZjRmXSBDb3VsZCBub3QgYXNzb2NpYXRlIGJyaWNrICdvdmlydDM4Mzk6L21udC9kYXRhL2Rp c3RyaWJ1dGVkJyBvZiB2b2x1bWUgJ2I4ODdiMDVlLTJlYTYtNDk2ZS05NTUyLTE1NWQ2NThlZWFh Nicgd2l0aCBjb3JyZWN0IG5ldHdvcmsgYXMgbm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGluIGNs dXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5OS05Y2QwLWI2M2U5MTliOWFjYScNCjIwMTYtMDYtMjMg MjI6MDg6NTksNjI0IFdBUk4gIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLmdsdXN0 ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9yWG1sUnBjXSAoRGVmYXVsdFF1YXJ0elNjaGVk dWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBub3QgYXNzb2NpYXRlIGJyaWNrICdvdmly dDM0MzU6L21udC9kYXRhL2lzbycgb2Ygdm9sdW1lICc4OWYzMjQ1Ny1jOGMzLTQ5MGUtYjQ5MS0x NmRkMjdkZTAwNzMnIHdpdGggY29ycmVjdCBuZXR3b3JrIGFzIG5vIGdsdXN0ZXIgbmV0d29yayBm b3VuZCBpbiBjbHVzdGVyICc3NWJkNjRkZS0wNGIyLTRhOTktOWNkMC1iNjNlOTE5YjlhY2EnDQoy MDE2LTA2LTIzIDIyOjA4OjU5LDYyNiBXQVJOICBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLnZkc2Jy b2tlci5nbHVzdGVyLkdsdXN0ZXJWb2x1bWVzTGlzdFJldHVybkZvclhtbFJwY10gKERlZmF1bHRR dWFydHpTY2hlZHVsZXJfV29ya2VyLTc2KSBbMWMxY2Y0Zl0gQ291bGQgbm90IGFzc29jaWF0ZSBi cmljayAnb3ZpcnQzNjM3Oi9tbnQvZGF0YS9pc28nIG9mIHZvbHVtZSAnODlmMzI0NTctYzhjMy00 OTBlLWI0OTEtMTZkZDI3ZGUwMDczJyB3aXRoIGNvcnJlY3QgbmV0d29yayBhcyBubyBnbHVzdGVy IG5ldHdvcmsgZm91bmQgaW4gY2x1c3RlciAnNzViZDY0ZGUtMDRiMi00YTk5LTljZDAtYjYzZTkx OWI5YWNhJw0KMjAxNi0wNi0yMyAyMjowODo1OSw2MjggV0FSTiAgW29yZy5vdmlydC5lbmdpbmUu Y29yZS52ZHNicm9rZXIuZ2x1c3Rlci5HbHVzdGVyVm9sdW1lc0xpc3RSZXR1cm5Gb3JYbWxScGNd IChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci03NikgWzFjMWNmNGZdIENvdWxkIG5vdCBh c3NvY2lhdGUgYnJpY2sgJ292aXJ0MzgzOTovbW50L2RhdGEvaXNvJyBvZiB2b2x1bWUgJzg5ZjMy NDU3LWM4YzMtNDkwZS1iNDkxLTE2ZGQyN2RlMDA3Mycgd2l0aCBjb3JyZWN0IG5ldHdvcmsgYXMg bm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5OS05 Y2QwLWI2M2U5MTliOWFjYScNCjIwMTYtMDYtMjMgMjI6MDg6NTksNjI5IElORk8gIFtvcmcub3Zp cnQuZW5naW5lLmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0VkRTQ29t bWFuZF0gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTc2KSBbMWMxY2Y0Zl0gRklOSVNI LCBHbHVzdGVyVm9sdW1lc0xpc3RWRFNDb21tYW5kLCByZXR1cm46IHtiODg3YjA1ZS0yZWE2LTQ5 NmUtOTU1Mi0xNTVkNjU4ZWVhYTY9b3JnLm92aXJ0LmVuZ2luZS5jb3JlLmNvbW1vbi5idXNpbmVz c2VudGl0aWVzLmdsdXN0ZXIuR2x1c3RlclZvbHVtZUVudGl0eUBkOGM5MDM5YiwgOTNlMzZjZGMt YWIxYi00MWVjLWFjN2YtOTY2Y2YzODU2YjU5PW9yZy5vdmlydC5lbmdpbmUuY29yZS5jb21tb24u YnVzaW5lc3NlbnRpdGllcy5nbHVzdGVyLkdsdXN0ZXJWb2x1bWVFbnRpdHlAZmU1YWIwMTksIDg5 ZjMyNDU3LWM4YzMtNDkwZS1iNDkxLTE2ZGQyN2RlMDA3Mz1vcmcub3ZpcnQuZW5naW5lLmNvcmUu Y29tbW9uLmJ1c2luZXNzZW50aXRpZXMuZ2x1c3Rlci5HbHVzdGVyVm9sdW1lRW50aXR5QDlhNTZk NjMzfSwgbG9nIGlkOiA0ODVhMDYxMQ0KMjAxNi0wNi0yMyAyMjowOTowNCw2NDUgSU5GTyAgW29y Zy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIuZ2x1c3Rlci5HbHVzdGVyVm9sdW1lc0xpc3RW RFNDb21tYW5kXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItODkpIFtdIFNUQVJULCBH bHVzdGVyVm9sdW1lc0xpc3RWRFNDb21tYW5kKEhvc3ROYW1lID0gb3ZpcnQzODM5LCBHbHVzdGVy Vm9sdW1lc0xpc3RWRFNQYXJhbWV0ZXJzOntydW5Bc3luYz0ndHJ1ZScsIGhvc3RJZD0nMzJjNTAw ZTUtMjY4ZC00MjZhLTlhNGEtMTA4NTM1ZTY3NzIyJ30pLCBsb2cgaWQ6IDQxYjY0NzlkDQo= --_000_CO2PR0801MB0743180FC142ED11D7D5D3E0A62E0CO2PR0801MB0743_ Content-Type: text/html; charset="utf-8" Content-ID: <D0C4FA3A5B1EE240A396E58D5F57A2E7@sct-15-1-485-2-msonline-outlook-1173a.templateTenant> Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iR2VuZXJhdG9yIiBjb250ZW50PSJNaWNyb3NvZnQgV29yZCAxNSAoZmlsdGVyZWQg bWVkaXVtKSI+DQo8c3R5bGU+PCEtLQ0KLyogRm9udCBEZWZpbml0aW9ucyAqLw0KQGZvbnQtZmFj ZQ0KCXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2 IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5vc2UtMToy IDE1IDUgMiAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3Jt YWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGluOw0KCW1hcmdpbi1i b3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJp IixzYW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXBy aW9yaXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQph OnZpc2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9sbG93ZWQNCgl7bXNvLXN0eWxlLXByaW9yaXR5 Ojk5Ow0KCWNvbG9yOiM5NTRGNzI7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQouTXNv Q2hwRGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTt9DQpAcGFnZSBXb3JkU2Vj dGlvbjENCgl7c2l6ZTo4LjVpbiAxMS4waW47DQoJbWFyZ2luOjEuMGluIDEuMGluIDEuMGluIDEu MGluO30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHls ZT4NCjwvaGVhZD4NCjxib2R5IGxhbmc9IkVOLVVTIiBsaW5rPSJibHVlIiB2bGluaz0iIzk1NEY3 MiI+DQo8ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+SGVs bG8sPC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj5IYXZlIDMgbm9kZXMgcnVubmluZyBib3RoIG9WaXJ0IGFuZCBHbHVz dGVyIG9uIDQgU1NE4oCZcyBlYWNoLiBBdCB0aGUgbW9tZW50LCB0aGVyZSBhcmUgdHdvIHBoeXNp Y2FsIG5pY3MsIG9uZSBoYXMgcHVibGljIGludGVybmV0IGFjY2VzcyBhbmQgdGhlIG90aGVyIGlz IGEgbm9uLXJvdXRhYmxlIG5ldHdvcmsgdXNlZCBmb3Igb3ZpcnRtZ210ICZhbXA7IGdsdXN0ZXIu PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8cCBjbGFz cz0iTXNvTm9ybWFsIj5JbiB0aGUgbG9naWNhbCBuZXR3b3JrcywgSSBoYXZlIHNlbGVjdGVkIGds dXN0ZXIgZm9yIHRoZSBub25yb3V0YWJsZSBuZXR3b3JrIHJ1bm5pbmcgb3ZpcnRtZ210IGFuZCBn bHVzdGVyIGhvd2V2ZXIsIHR3byBWTeKAmXMgcmFuZG9tbHkgcGF1c2UgZm9yIHdoYXQgc2VlbXMg bGlrZSBubyByZWFzb24uIFRoZXkgY2FuIGJvdGggYmUgcmVzdW1lZCB3aXRob3V0IGlzc3VlLjwv cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+T25lIHRlc3QgVk0gaGFzIDRHQiBvZiBtZW1vcnkgYW5kIGEgc21hbGwgZGlz ayDigJMgbm8gcHJvYmxlbXMgd2l0aCB0aGlzIG9uZS4gVHdvIG90aGVycyBoYXZlIDgwMEdCIGRp c2tzIGFuZCAzMkdCIG9mIFJBTSDigJMgYm90aCB2beKAmXMgZXhoaWJpdCB0aGUgc2FtZSBpc3N1 ZS48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBw dDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPjxvOnA+Jm5i c3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90Oyxz ZXJpZiI+SSBhbHNvIHNlZSB0aGVzZSBpbiB0aGUgb1ZpcnQgZGFzaGJvYXJkOjxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+PG86 cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7LHNlcmlmIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsNCjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+RmFpbGVkIHRvIHVwZGF0ZSBPVkYgZGlza3Mg OWU2MDMyOGQtMjlhZi00NTMzLTg0ZjktNjMzZDg3ZjU0OGE3LCBPVkYgZGF0YSBpc24ndCB1cGRh dGVkIG9uIHRob3NlIE9WRiBzdG9yZXMgKERhdGEgQ2VudGVyIHh4eHh4LCBTdG9yYWdlIERvbWFp biBzci12b2x1bWUwMSkuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMg TmV3IFJvbWFuJnF1b3Q7LHNlcmlmIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWls eTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPkp1biAyMywgMjAxNiA5OjU0OjAz IFBNPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7LHNlcmlmIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9y bWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1l cyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPlZEU00gY29tbWFuZCBmYWlsZWQ6IENvdWxkIG5vdCBh Y3F1aXJlIHJlc291cmNlLiBQcm9iYWJseSByZXNvdXJjZSBmYWN0b3J5IHRocmV3IGFuIGV4Y2Vw dGlvbi46ICgpPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJv bWFuJnF1b3Q7LHNlcmlmIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVv dDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPi8vLy8vLy8vLy8vLy8vLzxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+PG86 cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7LHNlcmlmIj5WTSB4eHh4eCBoYXMgYmVlbiBwYXVzZWQgZHVlIHRvIHVua25vd24gc3RvcmFn ZSBlcnJvci48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9t YW4mcXVvdDssc2VyaWYiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+Ly8vLy8vLy8vLy8vLy8vPG86cD48L286cD48 L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox Mi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LHNlcmlmIj48bzpw PiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVv dDssc2VyaWYiPkluIHRoZSBlcnJvciBsb2cgb24gdGhlIGVuZ2luZSwgSSBzZWUgdGhlc2U6PG86 cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LHNl cmlmIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcg Um9tYW4mcXVvdDssc2VyaWYiPkVSUk9SIFtvcmcub3ZpcnQuZW5naW5lLmNvcmUuZGFsLmRiYnJv a2VyLmF1ZGl0bG9naGFuZGxpbmcuQXVkaXRMb2dEaXJlY3Rvcl0gKEZvcmtKb2luUG9vbC0xLXdv cmtlci03KSBbMTBjYWY5M2VdIENvcnJlbGF0aW9uIElEOiBudWxsLCBDYWxsIFN0YWNrOiBudWxs LCBDdXN0b20gRXZlbnQgSUQ6DQogLTEsIE1lc3NhZ2U6IFZNIHh4eHh4eCBoYXMgYmVlbiBwYXVz ZWQgZHVlIHRvIHVua25vd24gc3RvcmFnZSBlcnJvci48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZh bWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPjxvOnA+Jm5ic3A7PC9vOnA+ PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6 MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+SU5G TyZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLmRhbC5kYmJyb2tlci5hdWRpdGxvZ2hhbmRs aW5nLkF1ZGl0TG9nRGlyZWN0b3JdIChGb3JrSm9pblBvb2wtMS13b3JrZXItMTEpIFsxMGNhZjkz ZV0gQ29ycmVsYXRpb24gSUQ6IG51bGwsIENhbGwgU3RhY2s6IG51bGwsIEN1c3RvbSBFdmVudCBJ RDoNCiAtMSwgTWVzc2FnZTogVk0geHh4eHh4IGhhcyByZWNvdmVyZWQgZnJvbSBwYXVzZWQgYmFj ayB0byB1cC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9t YW4mcXVvdDssc2VyaWYiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90 O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+Ly8vLy8vLy8vLy8vLy8vPG86cD48L286cD48 L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTox Mi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LHNlcmlmIj48bzpw PiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVv dDssc2VyaWYiPkhvc3RuYW1lcyBhcmUgYWxsIGxvY2FsIHRvIC9ldGMvaG9zdHMgb24gYWxsIHNl cnZlcnMg4oCTIHRoZXkgYWxzbyByZXNvbHZlIHdpdGhvdXQgaXNzdWUgZnJvbSBlYWNoIGhvc3Qu PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7 LHNlcmlmIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBO ZXcgUm9tYW4mcXVvdDssc2VyaWYiPi8vLy8vLy8vLy8vLy8vPG86cD48L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9u dC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LHNlcmlmIj48bzpwPiZuYnNwOzwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1z aXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYi PjIwMTYtMDYtMjMgMjI6MDg6NTksNjExIFdBUk4mbmJzcDsgW29yZy5vdmlydC5lbmdpbmUuY29y ZS52ZHNicm9rZXIuZ2x1c3Rlci5HbHVzdGVyVm9sdW1lc0xpc3RSZXR1cm5Gb3JYbWxScGNdIChE ZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci03NikgWzFjMWNmNGZdIENvdWxkIG5vdCBhc3Nv Y2lhdGUNCiBicmljayAnb3ZpcnQzNDM1Oi9tbnQvZGF0YS9zci12b2x1bWUwMScgb2Ygdm9sdW1l ICc5M2UzNmNkYy1hYjFiLTQxZWMtYWM3Zi05NjZjZjM4NTZiNTknIHdpdGggY29ycmVjdCBuZXR3 b3JrIGFzIG5vIGdsdXN0ZXIgbmV0d29yayBmb3VuZCBpbiBjbHVzdGVyICc3NWJkNjRkZS0wNGIy LTRhOTktOWNkMC1iNjNlOTE5YjlhY2EnPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1 b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LHNlcmlmIj4yMDE2LTA2LTIzIDIyOjA4OjU5LDYxNCBX QVJOJm5ic3A7IFtvcmcub3ZpcnQuZW5naW5lLmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3Rl clZvbHVtZXNMaXN0UmV0dXJuRm9yWG1sUnBjXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3Jr ZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBub3QgYXNzb2NpYXRlDQogYnJpY2sgJ292aXJ0MzYzNzov bW50L2RhdGEvc3Itdm9sdW1lMDEnIG9mIHZvbHVtZSAnOTNlMzZjZGMtYWIxYi00MWVjLWFjN2Yt OTY2Y2YzODU2YjU5JyB3aXRoIGNvcnJlY3QgbmV0d29yayBhcyBubyBnbHVzdGVyIG5ldHdvcmsg Zm91bmQgaW4gY2x1c3RlciAnNzViZDY0ZGUtMDRiMi00YTk5LTljZDAtYjYzZTkxOWI5YWNhJzxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90Oyxz ZXJpZiI+MjAxNi0wNi0yMyAyMjowODo1OSw2MTYgV0FSTiZuYnNwOyBbb3JnLm92aXJ0LmVuZ2lu ZS5jb3JlLnZkc2Jyb2tlci5nbHVzdGVyLkdsdXN0ZXJWb2x1bWVzTGlzdFJldHVybkZvclhtbFJw Y10gKERlZmF1bHRRdWFydHpTY2hlZHVsZXJfV29ya2VyLTc2KSBbMWMxY2Y0Zl0gQ291bGQgbm90 IGFzc29jaWF0ZQ0KIGJyaWNrICdvdmlydDM4Mzk6L21udC9kYXRhL3NyLXZvbHVtZTAxJyBvZiB2 b2x1bWUgJzkzZTM2Y2RjLWFiMWItNDFlYy1hYzdmLTk2NmNmMzg1NmI1OScgd2l0aCBjb3JyZWN0 IG5ldHdvcmsgYXMgbm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRl LTA0YjItNGE5OS05Y2QwLWI2M2U5MTliOWFjYSc8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWls eTomcXVvdDtUaW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPjIwMTYtMDYtMjMgMjI6MDg6NTks NjE4IFdBUk4mbmJzcDsgW29yZy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIuZ2x1c3Rlci5H bHVzdGVyVm9sdW1lc0xpc3RSZXR1cm5Gb3JYbWxScGNdIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVy X1dvcmtlci03NikgWzFjMWNmNGZdIENvdWxkIG5vdCBhc3NvY2lhdGUNCiBicmljayAnb3ZpcnQz NDM1Oi9tbnQvZGF0YS9kaXN0cmlidXRlZCcgb2Ygdm9sdW1lICdiODg3YjA1ZS0yZWE2LTQ5NmUt OTU1Mi0xNTVkNjU4ZWVhYTYnIHdpdGggY29ycmVjdCBuZXR3b3JrIGFzIG5vIGdsdXN0ZXIgbmV0 d29yayBmb3VuZCBpbiBjbHVzdGVyICc3NWJkNjRkZS0wNGIyLTRhOTktOWNkMC1iNjNlOTE5Yjlh Y2EnPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1 b3Q7LHNlcmlmIj4yMDE2LTA2LTIzIDIyOjA4OjU5LDYyMCBXQVJOJm5ic3A7IFtvcmcub3ZpcnQu ZW5naW5lLmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9y WG1sUnBjXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3Vs ZCBub3QgYXNzb2NpYXRlDQogYnJpY2sgJ292aXJ0MzYzNzovbW50L2RhdGEvZGlzdHJpYnV0ZWQn IG9mIHZvbHVtZSAnYjg4N2IwNWUtMmVhNi00OTZlLTk1NTItMTU1ZDY1OGVlYWE2JyB3aXRoIGNv cnJlY3QgbmV0d29yayBhcyBubyBnbHVzdGVyIG5ldHdvcmsgZm91bmQgaW4gY2x1c3RlciAnNzVi ZDY0ZGUtMDRiMi00YTk5LTljZDAtYjYzZTkxOWI5YWNhJzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQt ZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+MjAxNi0wNi0yMyAyMjow ODo1OSw2MjIgV0FSTiZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLnZkc2Jyb2tlci5nbHVz dGVyLkdsdXN0ZXJWb2x1bWVzTGlzdFJldHVybkZvclhtbFJwY10gKERlZmF1bHRRdWFydHpTY2hl ZHVsZXJfV29ya2VyLTc2KSBbMWMxY2Y0Zl0gQ291bGQgbm90IGFzc29jaWF0ZQ0KIGJyaWNrICdv dmlydDM4Mzk6L21udC9kYXRhL2Rpc3RyaWJ1dGVkJyBvZiB2b2x1bWUgJ2I4ODdiMDVlLTJlYTYt NDk2ZS05NTUyLTE1NWQ2NThlZWFhNicgd2l0aCBjb3JyZWN0IG5ldHdvcmsgYXMgbm8gZ2x1c3Rl ciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5OS05Y2QwLWI2M2U5 MTliOWFjYSc8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9t YW4mcXVvdDssc2VyaWYiPjIwMTYtMDYtMjMgMjI6MDg6NTksNjI0IFdBUk4mbmJzcDsgW29yZy5v dmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIuZ2x1c3Rlci5HbHVzdGVyVm9sdW1lc0xpc3RSZXR1 cm5Gb3JYbWxScGNdIChEZWZhdWx0UXVhcnR6U2NoZWR1bGVyX1dvcmtlci03NikgWzFjMWNmNGZd IENvdWxkIG5vdCBhc3NvY2lhdGUNCiBicmljayAnb3ZpcnQzNDM1Oi9tbnQvZGF0YS9pc28nIG9m IHZvbHVtZSAnODlmMzI0NTctYzhjMy00OTBlLWI0OTEtMTZkZDI3ZGUwMDczJyB3aXRoIGNvcnJl Y3QgbmV0d29yayBhcyBubyBnbHVzdGVyIG5ldHdvcmsgZm91bmQgaW4gY2x1c3RlciAnNzViZDY0 ZGUtMDRiMi00YTk5LTljZDAtYjYzZTkxOWI5YWNhJzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQtZmFt aWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+MjAxNi0wNi0yMyAyMjowODo1 OSw2MjYgV0FSTiZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLnZkc2Jyb2tlci5nbHVzdGVy LkdsdXN0ZXJWb2x1bWVzTGlzdFJldHVybkZvclhtbFJwY10gKERlZmF1bHRRdWFydHpTY2hlZHVs ZXJfV29ya2VyLTc2KSBbMWMxY2Y0Zl0gQ291bGQgbm90IGFzc29jaWF0ZQ0KIGJyaWNrICdvdmly dDM2Mzc6L21udC9kYXRhL2lzbycgb2Ygdm9sdW1lICc4OWYzMjQ1Ny1jOGMzLTQ5MGUtYjQ5MS0x NmRkMjdkZTAwNzMnIHdpdGggY29ycmVjdCBuZXR3b3JrIGFzIG5vIGdsdXN0ZXIgbmV0d29yayBm b3VuZCBpbiBjbHVzdGVyICc3NWJkNjRkZS0wNGIyLTRhOTktOWNkMC1iNjNlOTE5YjlhY2EnPG86 cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3IFJvbWFuJnF1b3Q7LHNl cmlmIj4yMDE2LTA2LTIzIDIyOjA4OjU5LDYyOCBXQVJOJm5ic3A7IFtvcmcub3ZpcnQuZW5naW5l LmNvcmUudmRzYnJva2VyLmdsdXN0ZXIuR2x1c3RlclZvbHVtZXNMaXN0UmV0dXJuRm9yWG1sUnBj XSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsxYzFjZjRmXSBDb3VsZCBub3Qg YXNzb2NpYXRlDQogYnJpY2sgJ292aXJ0MzgzOTovbW50L2RhdGEvaXNvJyBvZiB2b2x1bWUgJzg5 ZjMyNDU3LWM4YzMtNDkwZS1iNDkxLTE2ZGQyN2RlMDA3Mycgd2l0aCBjb3JyZWN0IG5ldHdvcmsg YXMgbm8gZ2x1c3RlciBuZXR3b3JrIGZvdW5kIGluIGNsdXN0ZXIgJzc1YmQ2NGRlLTA0YjItNGE5 OS05Y2QwLWI2M2U5MTliOWFjYSc8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtU aW1lcyBOZXcgUm9tYW4mcXVvdDssc2VyaWYiPjIwMTYtMDYtMjMgMjI6MDg6NTksNjI5IElORk8m bmJzcDsgW29yZy5vdmlydC5lbmdpbmUuY29yZS52ZHNicm9rZXIuZ2x1c3Rlci5HbHVzdGVyVm9s dW1lc0xpc3RWRFNDb21tYW5kXSAoRGVmYXVsdFF1YXJ0elNjaGVkdWxlcl9Xb3JrZXItNzYpIFsx YzFjZjRmXSBGSU5JU0gsIEdsdXN0ZXJWb2x1bWVzTGlzdFZEU0NvbW1hbmQsDQogcmV0dXJuOiB7 Yjg4N2IwNWUtMmVhNi00OTZlLTk1NTItMTU1ZDY1OGVlYWE2PW9yZy5vdmlydC5lbmdpbmUuY29y ZS5jb21tb24uYnVzaW5lc3NlbnRpdGllcy5nbHVzdGVyLkdsdXN0ZXJWb2x1bWVFbnRpdHlAZDhj OTAzOWIsIDkzZTM2Y2RjLWFiMWItNDFlYy1hYzdmLTk2NmNmMzg1NmI1OT1vcmcub3ZpcnQuZW5n aW5lLmNvcmUuY29tbW9uLmJ1c2luZXNzZW50aXRpZXMuZ2x1c3Rlci5HbHVzdGVyVm9sdW1lRW50 aXR5QGZlNWFiMDE5LCA4OWYzMjQ1Ny1jOGMzLTQ5MGUtYjQ5MS0xNmRkMjdkZTAwNzM9b3JnLm92 aXJ0LmVuZ2luZS5jb3JlLmNvbW1vbi5idXNpbmVzc2VudGl0aWVzLmdsdXN0ZXIuR2x1c3RlclZv bHVtZUVudGl0eUA5YTU2ZDYzM30sDQogbG9nIGlkOiA0ODVhMDYxMTxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0 O2ZvbnQtZmFtaWx5OiZxdW90O1RpbWVzIE5ldyBSb21hbiZxdW90OyxzZXJpZiI+MjAxNi0wNi0y MyAyMjowOTowNCw2NDUgSU5GTyZuYnNwOyBbb3JnLm92aXJ0LmVuZ2luZS5jb3JlLnZkc2Jyb2tl ci5nbHVzdGVyLkdsdXN0ZXJWb2x1bWVzTGlzdFZEU0NvbW1hbmRdIChEZWZhdWx0UXVhcnR6U2No ZWR1bGVyX1dvcmtlci04OSkgW10gU1RBUlQsIEdsdXN0ZXJWb2x1bWVzTGlzdFZEU0NvbW1hbmQo SG9zdE5hbWUNCiA9IG92aXJ0MzgzOSwgR2x1c3RlclZvbHVtZXNMaXN0VkRTUGFyYW1ldGVyczp7 cnVuQXN5bmM9J3RydWUnLCBob3N0SWQ9JzMyYzUwMGU1LTI2OGQtNDI2YS05YTRhLTEwODUzNWU2 NzcyMid9KSwgbG9nIGlkOiA0MWI2NDc5ZDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0K PC9ib2R5Pg0KPC9odG1sPg0K --_000_CO2PR0801MB0743180FC142ED11D7D5D3E0A62E0CO2PR0801MB0743_--
This is a multi-part message in MIME format. --------------070800090401040005080909 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Can you post the gluster mount logs from the node where paused VM was running (under /var/log/glusterfs/rhev-datarhev-data-center-mnt-glusterSD<mount-path>.log) ? Which version of glusterfs are you running? On 06/24/2016 07:49 AM, Bill Bill wrote:
Hello,
Have 3 nodes running both oVirt and Gluster on 4 SSDs each. At the moment, there are two physical nics, one has public internet access and the other is a non-routable network used for ovirtmgmt & gluster.
In the logical networks, I have selected gluster for the nonroutable network running ovirtmgmt and gluster however, two VMs randomly pause for what seems like no reason. They can both be resumed without issue.
One test VM has 4GB of memory and a small disk no problems with this one. Two others have 800GB disks and 32GB of RAM both vms exhibit the same issue.
I also see these in the oVirt dashboard:
Failed to update OVF disks 9e60328d-29af-4533-84f9-633d87f548a7, OVF data isn't updated on those OVF stores (Data Center xxxxx, Storage Domain sr-volume01).
Jun 23, 2016 9:54:03 PM
VDSM command failed: Could not acquire resource. Probably resource factory threw an exception.: ()
///////////////
VM xxxxx has been paused due to unknown storage error.
///////////////
In the error log on the engine, I see these:
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has been paused due to unknown storage error.
INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has recovered from paused back to up.
///////////////
Hostnames are all local to /etc/hosts on all servers they also resolve without issue from each host.
//////////////
2016-06-23 22:08:59,611 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,614 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,616 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,618 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,620 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,622 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,624 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,626 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,628 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,629 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] FINISH, GlusterVolumesListVDSCommand, return: {b887b05e-2ea6-496e-9552-155d658eeaa6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d8c9039b, 93e36cdc-ab1b-41ec-ac7f-966cf3856b59=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe5ab019, 89f32457-c8c3-490e-b491-16dd27de0073=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9a56d633}, log id: 485a0611
2016-06-23 22:09:04,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [] START, GlusterVolumesListVDSCommand(HostName = ovirt3839, GlusterVolumesListVDSParameters:{runAsync='true', hostId='32c500e5-268d-426a-9a4a-108535e67722'}), log id: 41b6479d
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------070800090401040005080909 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> Can you post the gluster mount logs from the node where paused VM was running (under /var/log/glusterfs/rhev-datarhev-data-center-mnt-glusterSD<mount-path>.log) ? <br> Which version of glusterfs are you running?<br> <br> <div class="moz-cite-prefix">On 06/24/2016 07:49 AM, Bill Bill wrote:<br> </div> <blockquote cite="mid:CO2PR0801MB0743180FC142ED11D7D5D3E0A62E0@CO2PR0801MB0743.namprd08.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class="WordSection1"> <p class="MsoNormal">Hello,</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Have 3 nodes running both oVirt and Gluster on 4 SSDs each. At the moment, there are two physical nics, one has public internet access and the other is a non-routable network used for ovirtmgmt & gluster.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">In the logical networks, I have selected gluster for the nonroutable network running ovirtmgmt and gluster however, two VMs randomly pause for what seems like no reason. They can both be resumed without issue.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">One test VM has 4GB of memory and a small disk no problems with this one. Two others have 800GB disks and 32GB of RAM both vms exhibit the same issue.</p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">I also see these in the oVirt dashboard:<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Failed to update OVF disks 9e60328d-29af-4533-84f9-633d87f548a7, OVF data isn't updated on those OVF stores (Data Center xxxxx, Storage Domain sr-volume01).<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Jun 23, 2016 9:54:03 PM<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">VDSM command failed: Could not acquire resource. Probably resource factory threw an exception.: ()<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">///////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">VM xxxxx has been paused due to unknown storage error.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">///////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">In the error log on the engine, I see these:<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has been paused due to unknown storage error.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has recovered from paused back to up.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">///////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Hostnames are all local to /etc/hosts on all servers they also resolve without issue from each host.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">//////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,611 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,614 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,616 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,618 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,620 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,622 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,624 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,626 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,628 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,629 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] FINISH, GlusterVolumesListVDSCommand, return: {b887b05e-2ea6-496e-9552-155d658eeaa6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d8c9039b, 93e36cdc-ab1b-41ec-ac7f-966cf3856b59=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe5ab019, 89f32457-c8c3-490e-b491-16dd27de0073=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9a56d633}, log id: 485a0611<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:09:04,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [] START, GlusterVolumesListVDSCommand(HostName = ovirt3839, GlusterVolumesListVDSParameters:{runAsync='true', hostId='32c500e5-268d-426a-9a4a-108535e67722'}), log id: 41b6479d<o:p></o:p></span></p> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------070800090401040005080909--
Hi Bill, After glusterfs 3.7.11, around 4-5 bugs were found in sharding and replicate modules and fixed, some of them causing the VM(s) to pause. Could you share the glusterfs client logs from around the time the issue was seen? This will help me confirm it's the same issue, or even debug further if this is a new issue. -Krutika On Fri, Jun 24, 2016 at 10:02 AM, Sahina Bose <sabose@redhat.com> wrote:
Can you post the gluster mount logs from the node where paused VM was running (under /var/log/glusterfs/rhev-datarhev-data-center-mnt-glusterSD<mount-path>.log) ? Which version of glusterfs are you running?
On 06/24/2016 07:49 AM, Bill Bill wrote:
Hello,
Have 3 nodes running both oVirt and Gluster on 4 SSD’s each. At the moment, there are two physical nics, one has public internet access and the other is a non-routable network used for ovirtmgmt & gluster.
In the logical networks, I have selected gluster for the nonroutable network running ovirtmgmt and gluster however, two VM’s randomly pause for what seems like no reason. They can both be resumed without issue.
One test VM has 4GB of memory and a small disk – no problems with this one. Two others have 800GB disks and 32GB of RAM – both vm’s exhibit the same issue.
I also see these in the oVirt dashboard:
Failed to update OVF disks 9e60328d-29af-4533-84f9-633d87f548a7, OVF data isn't updated on those OVF stores (Data Center xxxxx, Storage Domain sr-volume01).
Jun 23, 2016 9:54:03 PM
VDSM command failed: Could not acquire resource. Probably resource factory threw an exception.: ()
///////////////
VM xxxxx has been paused due to unknown storage error.
///////////////
In the error log on the engine, I see these:
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has been paused due to unknown storage error.
INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has recovered from paused back to up.
///////////////
Hostnames are all local to /etc/hosts on all servers – they also resolve without issue from each host.
//////////////
2016-06-23 22:08:59,611 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,614 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,616 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,618 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,620 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,622 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,624 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,626 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,628 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,629 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] FINISH, GlusterVolumesListVDSCommand, return: {b887b05e-2ea6-496e-9552-155d658eeaa6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d8c9039b, 93e36cdc-ab1b-41ec-ac7f-966cf3856b59=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe5ab019, 89f32457-c8c3-490e-b491-16dd27de0073=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9a56d633}, log id: 485a0611
2016-06-23 22:09:04,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [] START, GlusterVolumesListVDSCommand(HostName = ovirt3839, GlusterVolumesListVDSParameters:{runAsync='true', hostId='32c500e5-268d-426a-9a4a-108535e67722'}), log id: 41b6479d
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Bill Bill -
Krutika Dhananjay -
Sahina Bose