Hosted Engine/NFS Troubles
Hi, I'm having trouble with my hosted engine setup (v4.0) and could use some help. The problem I'm having is that whenever I try to add additional hosts to the setup via webadmin, the operation fails due to storage-related issues. webadmin shows the following error messages: "Host <host name> cannot access the Storage Domain(s) hosted_storage attached to the Data Center Default. Setting Host state to Non-Operational. Failed to connect Host ovirt-node-1 to Storage Pool Default" The VDSM log from the host shows the following error message: "Thread-18::ERROR::2017-07-17 13:01:11,483::sdc::146:: Storage.StorageDomainCache::(_findDomain) domain ca044720-e5cf-40a8-8b21-57a17026db7c not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'ca044720-e5cf-40a8-8b21-57a17026db7c',)" The engine log shows the following error messages: "2017-07-17 18:32:11,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-34) [] Domain 'ca044720-e5cf-40a8-8b21-57a17026db7c:hosted_storage' was reported with error code '358' 2017-07-17 18:32:11,410 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Storage Domain 'hosted_storage' of pool 'Default' is in problem in host 'ovirt-node-1' 2017-07-17 18:32:11,487 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host ovirt-node-1 reports about one of the Active Storage Domains as Problematic." I have ownership set to vdsm/kvm and full rwx rights enabled on both directories. I have successfully mounted both the master domain and the hosted_storage manually on one of the hosts I'm trying to add. I have attached the engine log and the VDSM log for that host. Could someone please help me figure out what's causing this? -Phillip Bailey
On Mon, Jul 17, 2017 at 9:05 PM, Phillip Bailey <phbailey@redhat.com> wrote:
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some help. The problem I'm having is that whenever I try to add additional hosts to the setup via webadmin, the operation fails due to storage-related issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage attached to the Data Center Default. Setting Host state to Non-Operational. Failed to connect Host ovirt-node-1 to Storage Pool Default"
Hi Phillip, your hosted engine storage is on nfs, right? Did you test if you can mount manually on each host? Luca -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca@gmail.com>
On Mon, Jul 17, 2017 at 3:34 PM, Luca 'remix_tj' Lorenzetto < lorenzetto.luca@gmail.com> wrote:
On Mon, Jul 17, 2017 at 9:05 PM, Phillip Bailey <phbailey@redhat.com> wrote:
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some help. The problem I'm having is that whenever I try to add additional hosts to the setup via webadmin, the operation fails due to storage-related issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage attached to the Data Center Default. Setting Host state to Non-Operational. Failed to connect Host ovirt-node-1 to Storage Pool Default"
Hi Phillip,
your hosted engine storage is on nfs, right? Did you test if you can mount manually on each host?
Hi Luca,
Yes, both storage domains are on NFS (v3) and I am able to successfully mount them manually on the hosts. Luca
-- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , < lorenzetto.luca@gmail.com>
--_000_25E6C4DA741949D8A77A14C928CF7722acroniscom_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 UGhpbGxpcCwNCg0KVGhlIHJlbGV2YW50IGxpbmVzIGZyb20gdGhlIHZkc20gbG9ncyBhcmUgdGhl IGZvbGxvd2luZzoNCg0KanNvbnJwYy5FeGVjdXRvci82OjpJTkZPOjoyMDE3LTA3LTE3IDE0OjI0 OjQxLDAwNTo6bG9nVXRpbHM6OjQ5OjpkaXNwYXRjaGVyOjood3JhcHBlcikgUnVuIGFuZCBwcm90 ZWN0OiBjb25uZWN0U3RvcmFnZVNlcnZlcihkb21UeXBlPTEsIHNwVVVJRD11JzAwMDAwMDAwLTAw MDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCcsIGNvbkxpc3Q9W3t1J3Byb3RvY29sDQpfdmVyc2lv bic6IDMsIHUnY29ubmVjdGlvbic6IHUnMTkyLjE2OC4xLjIxOi9zcnYvb3ZpcnQnLCB1J3VzZXIn OiB1J2t2bScsIHUnaWQnOiB1J2RiZWI4YWI0LTg0OWYtNDcyOC04ZWU5LWY4OTFiYjg0Y2UyZid9 XSwgb3B0aW9ucz1Ob25lKQ0KanNvbnJwYy5FeGVjdXRvci82OjpERUJVRzo6MjAxNy0wNy0xNyAx NDoyNDo0MSwwMDY6OmZpbGVVdGlsczo6MjA5OjpTdG9yYWdlLmZpbGVVdGlsczo6KGNyZWF0ZWRp cikgQ3JlYXRpbmcgZGlyZWN0b3J5OiAvcmhldi9kYXRhLWNlbnRlci9tbnQvMTkyLjE2OC4xLjIx Ol9zcnZfb3ZpcnQgbW9kZTogTm9uZQ0KanNvbnJwYy5FeGVjdXRvci82OjpERUJVRzo6MjAxNy0w Ny0xNyAxNDoyNDo0MSwwMDc6OmZpbGVVdGlsczo6MjE4OjpTdG9yYWdlLmZpbGVVdGlsczo6KGNy ZWF0ZWRpcikgVXNpbmcgZXhpc3RpbmcgZGlyZWN0b3J5OiAvcmhldi9kYXRhLWNlbnRlci9tbnQv MTkyLjE2OC4xLjIxOl9zcnZfb3ZpcnQNCmpzb25ycGMuRXhlY3V0b3IvNjo6SU5GTzo6MjAxNy0w Ny0xNyAxNDoyNDo0MSwwMDc6Om1vdW50OjoyMjY6OnN0b3JhZ2UuTW91bnQ6Oihtb3VudCkgbW91 bnRpbmcgMTkyLjE2OC4xLjIxOi9zcnYvb3ZpcnQgYXQgL3JoZXYvZGF0YS1jZW50ZXIvbW50LzE5 Mi4xNjguMS4yMTpfc3J2X292aXJ0DQpqc29ucnBjLkV4ZWN1dG9yLzY6OkVSUk9SOjoyMDE3LTA3 LTE3IDE0OjI2OjQ2LDA5ODo6aHNtOjoyNDAzOjpTdG9yYWdlLkhTTTo6KGNvbm5lY3RTdG9yYWdl U2VydmVyKSBDb3VsZCBub3QgY29ubmVjdCB0byBzdG9yYWdlU2VydmVyDQpUcmFjZWJhY2sgKG1v c3QgcmVjZW50IGNhbGwgbGFzdCk6DQogIEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9zdG9yYWdlL2hz bS5weSIsIGxpbmUgMjQwMCwgaW4gY29ubmVjdFN0b3JhZ2VTZXJ2ZXINCiAgICBjb25PYmouY29u bmVjdCgpDQogIEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9zdG9yYWdlL3N0b3JhZ2VTZXJ2ZXIucHki LCBsaW5lIDQ1NiwgaW4gY29ubmVjdA0KICAgIHJldHVybiBzZWxmLl9tb3VudENvbi5jb25uZWN0 KCkNCiAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL3N0b3JhZ2Uvc3RvcmFnZVNlcnZlci5weSIsIGxp bmUgMjM4LCBpbiBjb25uZWN0DQogICAgc2l4LnJlcmFpc2UodCwgdiwgdGIpDQogIEZpbGUgIi91 c3Ivc2hhcmUvdmRzbS9zdG9yYWdlL3N0b3JhZ2VTZXJ2ZXIucHkiLCBsaW5lIDIzMCwgaW4gY29u bmVjdA0KICAgIHNlbGYuX21vdW50Lm1vdW50KHNlbGYub3B0aW9ucywgc2VsZi5fdmZzVHlwZSwg Y2dyb3VwPXNlbGYuQ0dST1VQKQ0KICBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNr YWdlcy92ZHNtL3N0b3JhZ2UvbW91bnQucHkiLCBsaW5lIDIyOSwgaW4gbW91bnQNCiAgICB0aW1l b3V0PXRpbWVvdXQsIGNncm91cD1jZ3JvdXApDQogIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9z aXRlLXBhY2thZ2VzL3Zkc20vc3VwZXJ2ZHNtLnB5IiwgbGluZSA1MywgaW4gX19jYWxsX18NCiAg ICByZXR1cm4gY2FsbE1ldGhvZCgpDQogIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBh Y2thZ2VzL3Zkc20vc3VwZXJ2ZHNtLnB5IiwgbGluZSA1MSwgaW4gPGxhbWJkYT4NCiAgICAqKmt3 YXJncykNCiAgRmlsZSAiPHN0cmluZz4iLCBsaW5lIDIsIGluIG1vdW50DQogIEZpbGUgIi91c3Iv bGliNjQvcHl0aG9uMi43L211bHRpcHJvY2Vzc2luZy9tYW5hZ2Vycy5weSIsIGxpbmUgNzczLCBp biBfY2FsbG1ldGhvZA0KICAgIHJhaXNlIGNvbnZlcnRfdG9fZXJyb3Ioa2luZCwgcmVzdWx0KQ0K TW91bnRFcnJvcjogKDMyLCAnO21vdW50Lm5mczogQ29ubmVjdGlvbiB0aW1lZCBvdXRcbicpDQoN Cg0KRnJvbTogPHVzZXJzLWJvdW5jZXNAb3ZpcnQub3JnPiBvbiBiZWhhbGYgb2YgUGhpbGxpcCBC YWlsZXkgPHBoYmFpbGV5QHJlZGhhdC5jb20+DQpEYXRlOiBUdWVzZGF5LCAxOCBKdWx5IDIwMTcg YXQgMTM6NDgNClRvOiBMdWNhICdyZW1peF90aicgTG9yZW56ZXR0byA8bG9yZW56ZXR0by5sdWNh QGdtYWlsLmNvbT4NCkNjOiB1c2VycyA8dXNlcnNAb3ZpcnQub3JnPg0KU3ViamVjdDogUmU6IFtv dmlydC11c2Vyc10gSG9zdGVkIEVuZ2luZS9ORlMgVHJvdWJsZXMNCg0KT24gTW9uLCBKdWwgMTcs IDIwMTcgYXQgMzozNCBQTSwgTHVjYSAncmVtaXhfdGonIExvcmVuemV0dG8gPGxvcmVuemV0dG8u bHVjYUBnbWFpbC5jb208bWFpbHRvOmxvcmVuemV0dG8ubHVjYUBnbWFpbC5jb20+PiB3cm90ZToN Ck9uIE1vbiwgSnVsIDE3LCAyMDE3IGF0IDk6MDUgUE0sIFBoaWxsaXAgQmFpbGV5IDxwaGJhaWxl eUByZWRoYXQuY29tPG1haWx0bzpwaGJhaWxleUByZWRoYXQuY29tPj4gd3JvdGU6DQo+IEhpLA0K Pg0KPiBJJ20gaGF2aW5nIHRyb3VibGUgd2l0aCBteSBob3N0ZWQgZW5naW5lIHNldHVwICh2NC4w KSBhbmQgY291bGQgdXNlIHNvbWUNCj4gaGVscC4gVGhlIHByb2JsZW0gSSdtIGhhdmluZyBpcyB0 aGF0IHdoZW5ldmVyIEkgdHJ5IHRvIGFkZCBhZGRpdGlvbmFsIGhvc3RzDQo+IHRvIHRoZSBzZXR1 cCB2aWEgd2ViYWRtaW4sIHRoZSBvcGVyYXRpb24gZmFpbHMgZHVlIHRvIHN0b3JhZ2UtcmVsYXRl ZA0KPiBpc3N1ZXMuDQo+DQo+IHdlYmFkbWluIHNob3dzIHRoZSBmb2xsb3dpbmcgZXJyb3IgbWVz c2FnZXM6DQo+DQo+ICJIb3N0IDxob3N0IG5hbWU+IGNhbm5vdCBhY2Nlc3MgdGhlIFN0b3JhZ2Ug RG9tYWluKHMpIGhvc3RlZF9zdG9yYWdlDQo+IGF0dGFjaGVkIHRvIHRoZSBEYXRhIENlbnRlciBE ZWZhdWx0LiBTZXR0aW5nIEhvc3Qgc3RhdGUgdG8gTm9uLU9wZXJhdGlvbmFsLg0KPiBGYWlsZWQg dG8gY29ubmVjdCBIb3N0IG92aXJ0LW5vZGUtMSB0byBTdG9yYWdlIFBvb2wgRGVmYXVsdCINCj4N Cg0KSGkgUGhpbGxpcCwNCg0KeW91ciBob3N0ZWQgZW5naW5lIHN0b3JhZ2UgaXMgb24gbmZzLCBy aWdodD8gRGlkIHlvdSB0ZXN0IGlmIHlvdSBjYW4NCm1vdW50IG1hbnVhbGx5IG9uIGVhY2ggaG9z dD8NCkhpIEx1Y2EsDQoNClllcywgYm90aCBzdG9yYWdlIGRvbWFpbnMgYXJlIG9uIE5GUyAodjMp IGFuZCBJIGFtIGFibGUgdG8gc3VjY2Vzc2Z1bGx5IG1vdW50IHRoZW0gbWFudWFsbHkgb24gdGhl IGhvc3RzLg0KDQpMdWNhDQoNCg0KDQotLQ0KIkUnIGFzc3VyZG8gaW1waWVnYXJlIGdsaSB1b21p bmkgZGkgaW50ZWxsaWdlbnphIGVjY2VsbGVudGUgcGVyIGZhcmUNCmNhbGNvbGkgY2hlIHBvdHJl YmJlcm8gZXNzZXJlIGFmZmlkYXRpIGEgY2hpdW5xdWUgc2Ugc2kgdXNhc3Nlcm8gZGVsbGUNCm1h Y2NoaW5lIg0KR290dGZyaWVkIFdpbGhlbG0gdm9uIExlaWJuaXR6LCBGaWxvc29mbyBlIE1hdGVt YXRpY28gKDE2NDYtMTcxNikNCg0KIkludGVybmV0IMOoIGxhIHBpw7kgZ3JhbmRlIGJpYmxpb3Rl Y2EgZGVsIG1vbmRvLg0KTWEgaWwgcHJvYmxlbWEgw6ggY2hlIGkgbGlicmkgc29ubyB0dXR0aSBz cGFyc2kgc3VsIHBhdmltZW50byINCkpvaG4gQWxsZW4gUGF1bG9zLCBNYXRlbWF0aWNvICgxOTQ1 LXZpdmVudGUpDQoNCkx1Y2EgJ3JlbWl4X3RqJyBMb3JlbnpldHRvLCBodHRwOi8vd3d3LnJlbWl4 dGoubmV0ICwgPGxvcmVuemV0dG8ubHVjYUBnbWFpbC5jb208bWFpbHRvOmxvcmVuemV0dG8ubHVj YUBnbWFpbC5jb20+Pg0KDQo= --_000_25E6C4DA741949D8A77A14C928CF7722acroniscom_ Content-Type: text/html; charset="utf-8" Content-ID: <5C498FBBA2A36F47B6957EB3F4386884@acronis.com> Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iVGl0bGUiIGNvbnRlbnQ9IiI+DQo8bWV0YSBuYW1lPSJLZXl3b3JkcyIgY29udGVu dD0iIj4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg KGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8N CkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0 IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJ cGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8N CnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBjbTsN CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWls eToiVGltZXMgTmV3IFJvbWFuIjt9DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1z dHlsZS1wcmlvcml0eTo5OTsNCgljb2xvcjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxp bmU7fQ0KYTp2aXNpdGVkLCBzcGFuLk1zb0h5cGVybGlua0ZvbGxvd2VkDQoJe21zby1zdHlsZS1w cmlvcml0eTo5OTsNCgljb2xvcjpwdXJwbGU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9 DQpzcGFuLmhvZW56Yg0KCXttc28tc3R5bGUtbmFtZTpob2VuemI7fQ0Kc3Bhbi5FbWFpbFN0eWxl MTgNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7DQoJZm9udC1mYW1pbHk6Q2FsaWJy aTsNCgljb2xvcjp3aW5kb3d0ZXh0O30NCnNwYW4ubXNvSW5zDQoJe21zby1zdHlsZS10eXBlOmV4 cG9ydC1vbmx5Ow0KCW1zby1zdHlsZS1uYW1lOiIiOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxp bmU7DQoJY29sb3I6dGVhbDt9DQouTXNvQ2hwRGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBv cnQtb25seTsNCglmb250LXNpemU6MTAuMHB0O30NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXpl OjU5NS4wcHQgODQyLjBwdDsNCgltYXJnaW46Mi4wY20gNDIuNXB0IDIuMGNtIDMuMGNtO30NCmRp di5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT4NCjwvaGVh ZD4NCjxib2R5IGJnY29sb3I9IndoaXRlIiBsYW5nPSJFTi1HQiIgbGluaz0iYmx1ZSIgdmxpbms9 InB1cnBsZSI+DQo8ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+UGhpbGxpcCw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh bWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5U aGUgcmVsZXZhbnQgbGluZXMgZnJvbSB0aGUgdmRzbSBsb2dzIGFyZSB0aGUgZm9sbG93aW5nOjxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6 RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZh cmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPmpzb25ycGMuRXhlY3V0b3IvNjo6SU5GTzo6MjAxNy0wNy0x NyAxNDoyNDo0MSwwMDU6OmxvZ1V0aWxzOjo0OTo6ZGlzcGF0Y2hlcjo6KHdyYXBwZXIpIFJ1biBh bmQgcHJvdGVjdDogY29ubmVjdFN0b3JhZ2VTZXJ2ZXIoZG9tVHlwZT0xLCBzcFVVSUQ9dScwMDAw MDAwMC0wMDAwLTAwMDAtMDAwMC0wMDAwMDAwMDAwMDAnLA0KIGNvbkxpc3Q9W3t1J3Byb3RvY29s PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFn ZTpFTi1VUyI+X3ZlcnNpb24nOiAzLCB1J2Nvbm5lY3Rpb24nOiB1JzE5Mi4xNjguMS4yMTovc3J2 L292aXJ0JywgdSd1c2VyJzogdSdrdm0nLCB1J2lkJzogdSdkYmViOGFiNC04NDlmLTQ3MjgtOGVl OS1mODkxYmI4NGNlMmYnfV0sIG9wdGlvbnM9Tm9uZSk8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh bWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5qc29ucnBjLkV4ZWN1dG9y LzY6OkRFQlVHOjoyMDE3LTA3LTE3IDE0OjI0OjQxLDAwNjo6ZmlsZVV0aWxzOjoyMDk6OlN0b3Jh Z2UuZmlsZVV0aWxzOjooY3JlYXRlZGlyKSBDcmVhdGluZyBkaXJlY3Rvcnk6IC9yaGV2L2RhdGEt Y2VudGVyL21udC8xOTIuMTY4LjEuMjE6X3Nydl9vdmlydA0KIG1vZGU6IE5vbmU8bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5q c29ucnBjLkV4ZWN1dG9yLzY6OkRFQlVHOjoyMDE3LTA3LTE3IDE0OjI0OjQxLDAwNzo6ZmlsZVV0 aWxzOjoyMTg6OlN0b3JhZ2UuZmlsZVV0aWxzOjooY3JlYXRlZGlyKSBVc2luZyBleGlzdGluZyBk aXJlY3Rvcnk6IC9yaGV2L2RhdGEtY2VudGVyL21udC8xOTIuMTY4LjEuMjE6X3Nydl9vdmlydDxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6 RU4tVVMiPmpzb25ycGMuRXhlY3V0b3IvNjo6SU5GTzo6MjAxNy0wNy0xNyAxNDoyNDo0MSwwMDc6 Om1vdW50OjoyMjY6OnN0b3JhZ2UuTW91bnQ6Oihtb3VudCkgbW91bnRpbmcgMTkyLjE2OC4xLjIx Oi9zcnYvb3ZpcnQgYXQgL3JoZXYvZGF0YS1jZW50ZXIvbW50LzE5Mi4xNjguMS4yMTpfc3J2X292 aXJ0PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5n dWFnZTpFTi1VUyI+anNvbnJwYy5FeGVjdXRvci82OjpFUlJPUjo6MjAxNy0wNy0xNyAxNDoyNjo0 NiwwOTg6OmhzbTo6MjQwMzo6U3RvcmFnZS5IU006Oihjb25uZWN0U3RvcmFnZVNlcnZlcikgQ291 bGQgbm90IGNvbm5lY3QgdG8gc3RvcmFnZVNlcnZlcjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt aWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPlRyYWNlYmFjayAobW9zdCBy ZWNlbnQgY2FsbCBsYXN0KTo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9y bWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21z by1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgRmlsZSAmcXVvdDsvdXNyL3NoYXJlL3Zk c20vc3RvcmFnZS9oc20ucHkmcXVvdDssIGxpbmUgMjQwMCwgaW4gY29ubmVjdFN0b3JhZ2VTZXJ2 ZXI8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls ZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1 YWdlOkVOLVVTIj4mbmJzcDsgJm5ic3A7IGNvbk9iai5jb25uZWN0KCk8bzpwPjwvbzpwPjwvc3Bh bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw dDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsg RmlsZSAmcXVvdDsvdXNyL3NoYXJlL3Zkc20vc3RvcmFnZS9zdG9yYWdlU2VydmVyLnB5JnF1b3Q7 LCBsaW5lIDQ1NiwgaW4gY29ubmVjdDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGli cmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOyAmbmJzcDsgcmV0dXJuIHNlbGYu X21vdW50Q29uLmNvbm5lY3QoKTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7 bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOyBGaWxlICZxdW90Oy91c3Ivc2hhcmUv dmRzbS9zdG9yYWdlL3N0b3JhZ2VTZXJ2ZXIucHkmcXVvdDssIGxpbmUgMjM4LCBpbiBjb25uZWN0 PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFn ZTpFTi1VUyI+Jm5ic3A7ICZuYnNwOyBzaXgucmVyYWlzZSh0LCB2LCB0Yik8bzpwPjwvbzpwPjwv c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEx LjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJz cDsgRmlsZSAmcXVvdDsvdXNyL3NoYXJlL3Zkc20vc3RvcmFnZS9zdG9yYWdlU2VydmVyLnB5JnF1 b3Q7LCBsaW5lIDIzMCwgaW4gY29ubmVjdDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNh bGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOyAmbmJzcDsgc2VsZi5fbW91 bnQubW91bnQoc2VsZi5vcHRpb25zLCBzZWxmLl92ZnNUeXBlLCBjZ3JvdXA9c2VsZi5DR1JPVVAp PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFn ZTpFTi1VUyI+Jm5ic3A7IEZpbGUgJnF1b3Q7L3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2Fn ZXMvdmRzbS9zdG9yYWdlL21vdW50LnB5JnF1b3Q7LCBsaW5lIDIyOSwgaW4gbW91bnQ8bzpwPjwv bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1z aXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT Ij4mbmJzcDsgJm5ic3A7IHRpbWVvdXQ9dGltZW91dCwgY2dyb3VwPWNncm91cCk8bzpwPjwvbzpw Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4m bmJzcDsgRmlsZSAmcXVvdDsvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNtL3N1 cGVydmRzbS5weSZxdW90OywgbGluZSA1MywgaW4gX19jYWxsX188bzpwPjwvbzpwPjwvc3Bhbj48 L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtm b250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgJm5i c3A7IHJldHVybiBjYWxsTWV0aG9kKCk8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxp YnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgRmlsZSAmcXVvdDsvdXNyL2xp Yi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNtL3N1cGVydmRzbS5weSZxdW90OywgbGluZSA1 MSwgaW4gJmx0O2xhbWJkYSZndDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJp O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgJm5ic3A7ICoqa3dhcmdzKTxvOnA+ PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250 LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t VVMiPiZuYnNwOyBGaWxlICZxdW90OyZsdDtzdHJpbmcmZ3Q7JnF1b3Q7LCBsaW5lIDIsIGluIG1v dW50PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5n dWFnZTpFTi1VUyI+Jm5ic3A7IEZpbGUgJnF1b3Q7L3Vzci9saWI2NC9weXRob24yLjcvbXVsdGlw cm9jZXNzaW5nL21hbmFnZXJzLnB5JnF1b3Q7LCBsaW5lIDc3MywgaW4gX2NhbGxtZXRob2Q8bzpw PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9u dC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVO LVVTIj4mbmJzcDsgJm5ic3A7IHJhaXNlIGNvbnZlcnRfdG9fZXJyb3Ioa2luZCwgcmVzdWx0KTxv OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6 RU4tVVMiPk1vdW50RXJyb3I6ICgzMiwgJzttb3VudC5uZnM6DQo8Yj5Db25uZWN0aW9uIHRpbWVk IG91dDwvYj5cbicpPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFy ZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6 Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3Nw YW4+PC9wPg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjQjVDNERG IDEuMHB0O3BhZGRpbmc6My4wcHQgMGNtIDBjbSAwY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PGI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OkNhbGlicmk7Y29sb3I6YmxhY2siPkZyb206IDwv c3Bhbj4NCjwvYj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6Q2FsaWJyaTtjb2xvcjpibGFjayI+ Jmx0O3VzZXJzLWJvdW5jZXNAb3ZpcnQub3JnJmd0OyBvbiBiZWhhbGYgb2YgUGhpbGxpcCBCYWls ZXkgJmx0O3BoYmFpbGV5QHJlZGhhdC5jb20mZ3Q7PGJyPg0KPGI+RGF0ZTogPC9iPlR1ZXNkYXks IDE4IEp1bHkgMjAxNyBhdCAxMzo0ODxicj4NCjxiPlRvOiA8L2I+THVjYSAncmVtaXhfdGonIExv cmVuemV0dG8gJmx0O2xvcmVuemV0dG8ubHVjYUBnbWFpbC5jb20mZ3Q7PGJyPg0KPGI+Q2M6IDwv Yj51c2VycyAmbHQ7dXNlcnNAb3ZpcnQub3JnJmd0Ozxicj4NCjxiPlN1YmplY3Q6IDwvYj5SZTog W292aXJ0LXVzZXJzXSBIb3N0ZWQgRW5naW5lL05GUyBUcm91YmxlczxvOnA+PC9vOnA+PC9zcGFu PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9v OnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPGRpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij5PbiBNb24sIEp1bCAxNywgMjAxNyBhdCAzOjM0IFBNLCBMdWNhICdyZW1peF90aicgTG9yZW56 ZXR0byAmbHQ7PGEgaHJlZj0ibWFpbHRvOmxvcmVuemV0dG8ubHVjYUBnbWFpbC5jb20iIHRhcmdl dD0iX2JsYW5rIj5sb3JlbnpldHRvLmx1Y2FAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PG86cD48 L286cD48L3A+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29s aWQgI0NDQ0NDQyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQu OHB0O21hcmdpbi1yaWdodDowY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdp bi1ib3R0b206MTIuMHB0Ij5PbiBNb24sIEp1bCAxNywgMjAxNyBhdCA5OjA1IFBNLCBQaGlsbGlw IEJhaWxleSAmbHQ7PGEgaHJlZj0ibWFpbHRvOnBoYmFpbGV5QHJlZGhhdC5jb20iPnBoYmFpbGV5 QHJlZGhhdC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+DQomZ3Q7IEhpLDxicj4NCiZndDs8YnI+DQom Z3Q7IEknbSBoYXZpbmcgdHJvdWJsZSB3aXRoIG15IGhvc3RlZCBlbmdpbmUgc2V0dXAgKHY0LjAp IGFuZCBjb3VsZCB1c2Ugc29tZTxicj4NCiZndDsgaGVscC4gVGhlIHByb2JsZW0gSSdtIGhhdmlu ZyBpcyB0aGF0IHdoZW5ldmVyIEkgdHJ5IHRvIGFkZCBhZGRpdGlvbmFsIGhvc3RzPGJyPg0KJmd0 OyB0byB0aGUgc2V0dXAgdmlhIHdlYmFkbWluLCB0aGUgb3BlcmF0aW9uIGZhaWxzIGR1ZSB0byBz dG9yYWdlLXJlbGF0ZWQ8YnI+DQomZ3Q7IGlzc3Vlcy48YnI+DQomZ3Q7PGJyPg0KJmd0OyB3ZWJh ZG1pbiBzaG93cyB0aGUgZm9sbG93aW5nIGVycm9yIG1lc3NhZ2VzOjxicj4NCiZndDs8YnI+DQom Z3Q7ICZxdW90O0hvc3QgJmx0O2hvc3QgbmFtZSZndDsgY2Fubm90IGFjY2VzcyB0aGUgU3RvcmFn ZSBEb21haW4ocykgaG9zdGVkX3N0b3JhZ2U8YnI+DQomZ3Q7IGF0dGFjaGVkIHRvIHRoZSBEYXRh IENlbnRlciBEZWZhdWx0LiBTZXR0aW5nIEhvc3Qgc3RhdGUgdG8gTm9uLU9wZXJhdGlvbmFsLjxi cj4NCiZndDsgRmFpbGVkIHRvIGNvbm5lY3QgSG9zdCBvdmlydC1ub2RlLTEgdG8gU3RvcmFnZSBQ b29sIERlZmF1bHQmcXVvdDs8YnI+DQomZ3Q7PGJyPg0KPGJyPg0KSGkgUGhpbGxpcCw8YnI+DQo8 YnI+DQp5b3VyIGhvc3RlZCBlbmdpbmUgc3RvcmFnZSBpcyBvbiBuZnMsIHJpZ2h0PyBEaWQgeW91 IHRlc3QgaWYgeW91IGNhbjxicj4NCm1vdW50IG1hbnVhbGx5IG9uIGVhY2ggaG9zdD88bzpwPjwv bzpwPjwvcD4NCjwvYmxvY2txdW90ZT4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5IaSBM dWNhLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs Ij5ZZXMsIGJvdGggc3RvcmFnZSBkb21haW5zIGFyZSBvbiBORlMgKHYzKSBhbmQgSSBhbSBhYmxl IHRvIHN1Y2Nlc3NmdWxseSBtb3VudCB0aGVtIG1hbnVhbGx5IG9uIHRoZSBob3N0cy48bzpwPjwv bzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7 PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVy LWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdp bi1sZWZ0OjQuOHB0O21hcmdpbi1yaWdodDowY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gY2xhc3M9ImhvZW56YiI+PHNwYW4gc3R5bGU9ImNvbG9yOiM4ODg4ODgiPkx1Y2E8L3NwYW4+ PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjojODg4ODg4Ij48YnI+DQo8YnI+DQo8YnI+DQo8YnI+ DQo8c3BhbiBjbGFzcz0iaG9lbnpiIj4tLTwvc3Bhbj48YnI+DQo8c3BhbiBjbGFzcz0iaG9lbnpi Ij4mcXVvdDtFJyBhc3N1cmRvIGltcGllZ2FyZSBnbGkgdW9taW5pIGRpIGludGVsbGlnZW56YSBl Y2NlbGxlbnRlIHBlciBmYXJlPC9zcGFuPjxicj4NCjxzcGFuIGNsYXNzPSJob2VuemIiPmNhbGNv bGkgY2hlIHBvdHJlYmJlcm8gZXNzZXJlIGFmZmlkYXRpIGEgY2hpdW5xdWUgc2Ugc2kgdXNhc3Nl cm8gZGVsbGU8L3NwYW4+PGJyPg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+bWFjY2hpbmUmcXVvdDs8 L3NwYW4+PGJyPg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+R290dGZyaWVkIFdpbGhlbG0gdm9uIExl aWJuaXR6LCBGaWxvc29mbyBlIE1hdGVtYXRpY28gKDE2NDYtMTcxNik8L3NwYW4+PGJyPg0KPGJy Pg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+JnF1b3Q7SW50ZXJuZXQgw6ggbGEgcGnDuSBncmFuZGUg YmlibGlvdGVjYSBkZWwgbW9uZG8uPC9zcGFuPjxicj4NCjxzcGFuIGNsYXNzPSJob2VuemIiPk1h IGlsIHByb2JsZW1hIMOoIGNoZSBpIGxpYnJpIHNvbm8gdHV0dGkgc3BhcnNpIHN1bCBwYXZpbWVu dG8mcXVvdDs8L3NwYW4+PGJyPg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+Sm9obiBBbGxlbiBQYXVs b3MsIE1hdGVtYXRpY28gKDE5NDUtdml2ZW50ZSk8L3NwYW4+PGJyPg0KPGJyPg0KPHNwYW4gY2xh c3M9ImhvZW56YiI+THVjYSAncmVtaXhfdGonIExvcmVuemV0dG8sIDxhIGhyZWY9Imh0dHA6Ly93 d3cucmVtaXh0ai5uZXQiIHRhcmdldD0iX2JsYW5rIj4NCmh0dHA6Ly93d3cucmVtaXh0ai5uZXQ8 L2E+ICwgJmx0OzxhIGhyZWY9Im1haWx0bzpsb3JlbnpldHRvLmx1Y2FAZ21haWwuY29tIj5sb3Jl bnpldHRvLmx1Y2FAZ21haWwuY29tPC9hPiZndDs8L3NwYW4+PC9zcGFuPjxvOnA+PC9vOnA+PC9w Pg0KPC9ibG9ja3F1b3RlPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNw OzwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg== --_000_25E6C4DA741949D8A77A14C928CF7722acroniscom_--
On Tue, Jul 18, 2017 at 7:09 AM, Pavel Gashev <Pax@acronis.com> wrote:
Phillip,
The relevant lines from the vdsm logs are the following:
jsonrpc.Executor/6::INFO::2017-07-17 14:24:41,005::logUtils::49::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'protocol
_version': 3, u'connection': u'192.168.1.21:/srv/ovirt', u'user': u'kvm', u'id': u'dbeb8ab4-849f-4728-8ee9-f891bb84ce2f'}], options=None)
jsonrpc.Executor/6::DEBUG::2017-07-17 14:24:41,006::fileUtils::209::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/mnt/192.168.1.21:_srv_ovirt mode: None
jsonrpc.Executor/6::DEBUG::2017-07-17 14:24:41,007::fileUtils::218::Storage.fileUtils::(createdir) Using existing directory: /rhev/data-center/mnt/192.168.1.21:_srv_ovirt
jsonrpc.Executor/6::INFO::2017-07-17 14:24:41,007::mount::226::storage.Mount::(mount) mounting 192.168.1.21:/srv/ovirt at /rhev/data-center/mnt/192.168. 1.21:_srv_ovirt
jsonrpc.Executor/6::ERROR::2017-07-17 14:26:46,098::hsm::2403:: Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2400, in connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 456, in connect
return self._mountCon.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 238, in connect
six.reraise(t, v, tb)
File "/usr/share/vdsm/storage/storageServer.py", line 230, in connect
self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 229, in mount
timeout=timeout, cgroup=cgroup)
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__
return callMethod()
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda>
**kwargs)
File "<string>", line 2, in mount
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
raise convert_to_error(kind, result)
MountError: (32, ';mount.nfs: *Connection timed out*\n')
I saw this as well, but I don't understand why I don't have the same problem when mounting the domain manually and writing to it as the vdsm user. Thanks for pointing it out, though. I should have included that in the original message.
*From: *<users-bounces@ovirt.org> on behalf of Phillip Bailey < phbailey@redhat.com> *Date: *Tuesday, 18 July 2017 at 13:48 *To: *Luca 'remix_tj' Lorenzetto <lorenzetto.luca@gmail.com> *Cc: *users <users@ovirt.org> *Subject: *Re: [ovirt-users] Hosted Engine/NFS Troubles
On Mon, Jul 17, 2017 at 3:34 PM, Luca 'remix_tj' Lorenzetto < lorenzetto.luca@gmail.com> wrote:
On Mon, Jul 17, 2017 at 9:05 PM, Phillip Bailey <phbailey@redhat.com> wrote:
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some help. The problem I'm having is that whenever I try to add additional hosts to the setup via webadmin, the operation fails due to storage-related issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage attached to the Data Center Default. Setting Host state to Non-Operational. Failed to connect Host ovirt-node-1 to Storage Pool Default"
Hi Phillip,
your hosted engine storage is on nfs, right? Did you test if you can mount manually on each host?
Hi Luca,
Yes, both storage domains are on NFS (v3) and I am able to successfully mount them manually on the hosts.
Luca
-- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , < lorenzetto.luca@gmail.com>
--_000_150037588204129266leedsbeckettacuk_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I used the troubleshooting guide at documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/ and exported using (rw,sync,no_subtree_check,all_squash,anonuid=3D36,anongid=3D36) This was for the DATA_DOMAIN not hosted storage but it may help. Regards, Paul S. ________________________________ From: users-bounces@ovirt.org <users-bounces@ovirt.org> on behalf of Philli= p Bailey <phbailey@redhat.com> Sent: 17 July 2017 20:05 To: users Subject: [ovirt-users] Hosted Engine/NFS Troubles Hi, I'm having trouble with my hosted engine setup (v4.0) and could use some he= lp. The problem I'm having is that whenever I try to add additional hosts t= o the setup via webadmin, the operation fails due to storage-related issues= . webadmin shows the following error messages: "Host <host name> cannot access the Storage Domain(s) hosted_storage attach= ed to the Data Center Default. Setting Host state to Non-Operational. Failed to connect Host ovirt-node-1 to Storage Pool Default" The VDSM log from the host shows the following error message: "Thread-18::ERROR::2017-07-17 13:01:11,483::sdc::146::Storage.StorageDomain= Cache::(_findDomain) domain ca044720-e5cf-40a8-8b21-57a17026db7c not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'ca044720-e5cf-= 40a8-8b21-57a17026db7c',)" The engine log shows the following error messages: "2017-07-17 18:32:11,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.I= rsProxyData] (org.ovirt.thread.pool-6-thread-34) [] Domain 'ca044720-e5cf-4= 0a8-8b21-57a17026db7c:hosted_storage' was reported with error code '358' 2017-07-17 18:32:11,410 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand= ] (org.ovirt.thread.pool-6-thread-34) [] Storage Domain 'hosted_storage' of= pool 'Default' is in problem in host 'ovirt-node-1' 2017-07-17 18:32:11,487 ERROR [org.ovirt.engine.core.dal.dbbroker.auditlogh= andling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] Correlatio= n ID: null, Call Stack: null, Custom Event ID: -1, Message: Host ovirt-node= -1 reports about one of the Active Storage Domains as Problematic." I have ownership set to vdsm/kvm and full rwx rights enabled on both direct= ories. I have successfully mounted both the master domain and the hosted_st= orage manually on one of the hosts I'm trying to add. I have attached the e= ngine log and the VDSM log for that host. Could someone please help me figure out what's causing this? -Phillip Bailey To view the terms under which this email is distributed, please go to:- http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html --_000_150037588204129266leedsbeckettacuk_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none"><!--P{margin-top:0;margin-b= ottom:0;} --></style> </head> <body dir=3D"ltr" style=3D"font-size:12pt;color:#000000;background-color:#F= FFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"> <p>I used the troubleshooting guide at </p> <p><br> </p> <p>documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/= </p> <p><br> </p> <p>and exported using</p> <p><br> </p> <p>(rw,sync,no_subtree_check,all_squash,anonuid=3D36,anongid=3D36) </p> <p><br> </p> <p>This was for the DATA_DOMAIN not hosted storage but it may help.</p> <p><br> </p> <p>Regards,</p> <p><br> </p> <p>Paul S.<br> </p> <div style=3D"color: rgb(33, 33, 33);"> <hr tabindex=3D"-1" style=3D"display:inline-block; width:98%"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font style=3D"font-size:11pt" face= =3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b> users-bounces@ovirt= .org <users-bounces@ovirt.org> on behalf of Phillip Bailey <phbail= ey@redhat.com><br> <b>Sent:</b> 17 July 2017 20:05<br> <b>To:</b> users<br> <b>Subject:</b> [ovirt-users] Hosted Engine/NFS Troubles</font> <div> </div> </div> <div> <div dir=3D"ltr"> <div><span style=3D"font-size:12.8px">Hi,</span> <div style=3D"font-size:12.8px"><br> </div> <div style=3D"font-size:12.8px">I'm having trouble with my hosted engine se= tup (v4.0) and could use some help. <span style=3D"font-size:12.8px">T= he problem I'm having is that whenever I try to add additional hosts to the= setup via webadmin, the operation fails due to storage-related issues. </span></div> <div style=3D"font-size:12.8px"><br> webadmin shows the following error messages:</div> <div style=3D"font-size:12.8px"><br> </div> <div style=3D"font-size:12.8px">"Host <host name> cannot access = the Storage Domain(s) hosted_storage attached to the Data Center Default. S= etting Host state to Non-Operational.</div> <div style=3D"font-size:12.8px"><span style=3D"font-size:12.8px">Failed to = connect Host ovirt-node-1 to Storage Pool Default"<br> </span><br> </div> <div style=3D"font-size:12.8px"><br> </div> <div style=3D"font-size:12.8px">The VDSM log from the host shows the follow= ing error message:</div> <div style=3D"font-size:12.8px"><br> </div> <div style=3D"font-size:12.8px">"<span style=3D"font-size:12.8px">Thre= ad-18::ERROR::2017-07-17 13:01:11,483::sdc::146::<wbr>Storage.StorageDomain= Cache::(_<wbr>findDomain) domain ca044720-e5cf-40a8-8b21-<wbr>57a17026db7c = not found</span></div> <div style=3D"font-size:12.8px">Traceback (most recent call last):</div> <div style=3D"font-size:12.8px"> File "/usr/share/vdsm/storage/s= dc.<wbr>py", line 144, in _findDomain</div> <div style=3D"font-size:12.8px"> dom =3D findMethod(sdUUID)</d= iv> <div style=3D"font-size:12.8px"> File "/usr/share/vdsm/storage/s= dc.<wbr>py", line 174, in _findUnfetchedDomain</div> <div style=3D"font-size:12.8px"> raise se.StorageDomainDoesNot= Exist(<wbr>sdUUID)</div> <div style=3D"font-size:12.8px">StorageDomainDoesNotExist: Storage domain d= oes not exist: (u'ca044720-e5cf-40a8-8b21-<wbr>57a17026db7c',)"</div> <div style=3D"font-size:12.8px"><br> <br> </div> <div style=3D"font-size:12.8px">The engine log shows the following error me= ssages:<br> <br> <div style=3D"font-size:12.8px">"2017-07-17 18:32:11,409 ERROR [org.ov= irt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>IrsProxyData] (org.ovirt.thre= ad.pool-6-<wbr>thread-34) [] Domain 'ca044720-e5cf-40a8-8b21-<wbr>57a17026d= b7c:hosted_storage' was reported with error code '358'</div> <div style=3D"font-size:12.8px">2017-07-17 18:32:11,410 ERROR [org.ovirt.en= gine.core.bll.<wbr>InitVdsOnUpCommand] (org.ovirt.thread.pool-6-<wbr>thread= -34) [] Storage Domain 'hosted_storage' of pool 'Default' is in problem in = host 'ovirt-node-1'</div> <div style=3D"font-size:12.8px">2017-07-17 18:32:11,487 ERROR [org.ovirt.en= gine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ov= irt.thread.pool-6-<wbr>thread-34) [] Correlation ID: null, Call Stack: null= , Custom Event ID: -1, Message: Host ovirt-node-1 reports about one of the Active Storage Domains as Problemati= c."</div> </div> <div style=3D"font-size:12.8px"><br> </div> <div style=3D"font-size:12.8px"><br> </div> <div style=3D"font-size:12.8px">I have ownership set to vdsm/kvm and full r= wx rights enabled on both directories. I have successfully mounted both the= master domain and the hosted_storage manually on one of the hosts I'm tryi= ng to add. I have attached the engine log and the VDSM log for that host.<br> <br> Could someone please help me figure out what's causing this?</div> </div> <div><br> </div> <div> <div class=3D"gmail_signature"> <div dir=3D"ltr">-Phillip Bailey</div> </div> </div> </div> </div> </div> To view the terms under which this email is distributed, please go to:- <br=
<a href=3D"http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html"= target=3D"_blank">http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaim= er.html</a> <p></p> </body> </html> --_000_150037588204129266leedsbeckettacuk_--
I used the troubleshooting guide at
documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/
and exported using
(rw,sync,no_subtree_check,all_squash,anonuid=36,anongid=36)
This was for the DATA_DOMAIN not hosted storage but it may help.
My exports file has the same line, so I don't believe that's the problem. I went through the recommended steps on the troubleshooting guide and everything looks to be fine, but the problem still exists. The
On Tue, Jul 18, 2017 at 7:04 AM, Staniforth, Paul < P.Staniforth@leedsbeckett.ac.uk> wrote: troubleshooting guide never came up in my previous googling of the problem though, so thanks for pointing me to that. =)
Regards,
Paul S. ------------------------------ *From:* users-bounces@ovirt.org <users-bounces@ovirt.org> on behalf of Phillip Bailey <phbailey@redhat.com> *Sent:* 17 July 2017 20:05 *To:* users *Subject:* [ovirt-users] Hosted Engine/NFS Troubles
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some help. The problem I'm having is that whenever I try to add additional hosts to the setup via webadmin, the operation fails due to storage-related issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage attached to the Data Center Default. Setting Host state to Non-Operational. Failed to connect Host ovirt-node-1 to Storage Pool Default"
The VDSM log from the host shows the following error message:
"Thread-18::ERROR::2017-07-17 13:01:11,483::sdc::146::Storag e.StorageDomainCache::(_findDomain) domain ca044720-e5cf-40a8-8b21-57a17026db7c not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'ca044720-e5cf-40a8-8b21-57a17026db7c',)"
The engine log shows the following error messages:
"2017-07-17 18:32:11,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-6-thread-34) [] Domain 'ca044720-e5cf-40a8-8b21-57a17026db7c:hosted_storage' was reported with error code '358' 2017-07-17 18:32:11,410 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-34) [] Storage Domain 'hosted_storage' of pool 'Default' is in problem in host 'ovirt-node-1' 2017-07-17 18:32:11,487 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host ovirt-node-1 reports about one of the Active Storage Domains as Problematic."
I have ownership set to vdsm/kvm and full rwx rights enabled on both directories. I have successfully mounted both the master domain and the hosted_storage manually on one of the hosts I'm trying to add. I have attached the engine log and the VDSM log for that host.
Could someone please help me figure out what's causing this?
-Phillip Bailey To view the terms under which this email is distributed, please go to:- http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
participants (4)
-
Luca 'remix_tj' Lorenzetto -
Pavel Gashev -
Phillip Bailey -
Staniforth, Paul