3:05 p.m.
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some
help. The problem I'm having is that whenever I try to add additional hosts
to the setup via webadmin, the operation fails due to storage-related
issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage
attached to the Data Center Default. Setting Host state to Non-Operational.
Failed to connect Host ovirt-node-1 to Storage Pool Default"
The VDSM log from the host shows the following error message:
"Thread-18::ERROR::2017-07-17 13:01:11,483::sdc::146::
Storage.StorageDomainCache::(_findDomain) domain
ca044720-e5cf-40a8-8b21-57a17026db7c
not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'ca044720-e5cf-40a8-8b21-57a17026db7c',)"
The engine log shows the following error messages:
"2017-07-17 18:32:11,409 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-6-thread-34) [] Domain
'ca044720-e5cf-40a8-8b21-57a17026db7c:hosted_storage'
was reported with error code '358'
2017-07-17 18:32:11,410 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(org.ovirt.thread.pool-6-thread-34) [] Storage Domain 'hosted_storage' of
pool 'Default' is in problem in host 'ovirt-node-1'
2017-07-17 18:32:11,487 ERROR [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34)
[] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message:
Host ovirt-node-1 reports about one of the Active Storage Domains as
Problematic."
I have ownership set to vdsm/kvm and full rwx rights enabled on both
directories. I have successfully mounted both the master domain and the
hosted_storage manually on one of the hosts I'm trying to add. I have
attached the engine log and the VDSM log for that host.
Could someone please help me figure out what's causing this?
-Phillip Bailey
3:34 p.m.
On Mon, Jul 17, 2017 at 9:05 PM, Phillip Bailey <phbailey(a)redhat.com> wrote:
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some
help. The problem I'm having is that whenever I try to add additional hosts
to the setup via webadmin, the operation fails due to storage-related
issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage
attached to the Data Center Default. Setting Host state to Non-Operational.
Failed to connect Host ovirt-node-1 to Storage Pool Default"
Hi Phillip,
your hosted engine storage is on nfs, right? Did you test if you can
mount manually on each host?
Luca
--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
<lorenzetto.luca(a)gmail.com>
6:48 a.m.
On Mon, Jul 17, 2017 at 3:34 PM, Luca 'remix_tj' Lorenzetto <
lorenzetto.luca(a)gmail.com> wrote:
On Mon, Jul 17, 2017 at 9:05 PM, Phillip Bailey
<phbailey(a)redhat.com>
wrote:
> Hi,
>
> I'm having trouble with my hosted engine setup (v4.0) and could use some
> help. The problem I'm having is that whenever I try to add additional
hosts
> to the setup via webadmin, the operation fails due to storage-related
> issues.
>
> webadmin shows the following error messages:
>
> "Host <host name> cannot access the Storage Domain(s) hosted_storage
> attached to the Data Center Default. Setting Host state to
Non-Operational.
> Failed to connect Host ovirt-node-1 to Storage Pool Default"
>
Hi Phillip,
your hosted engine storage is on nfs, right? Did you test if you can
mount manually on each host?
Hi Luca,
Yes, both storage domains are on NFS (v3) and I am able to successfully
mount them manually on the hosts.
Luca
--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <
lorenzetto.luca(a)gmail.com>
7:09 a.m.
--_000_25E6C4DA741949D8A77A14C928CF7722acroniscom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
UGhpbGxpcCwNCg0KVGhlIHJlbGV2YW50IGxpbmVzIGZyb20gdGhlIHZkc20gbG9ncyBhcmUgdGhl
IGZvbGxvd2luZzoNCg0KanNvbnJwYy5FeGVjdXRvci82OjpJTkZPOjoyMDE3LTA3LTE3IDE0OjI0
OjQxLDAwNTo6bG9nVXRpbHM6OjQ5OjpkaXNwYXRjaGVyOjood3JhcHBlcikgUnVuIGFuZCBwcm90
ZWN0OiBjb25uZWN0U3RvcmFnZVNlcnZlcihkb21UeXBlPTEsIHNwVVVJRD11JzAwMDAwMDAwLTAw
MDAtMDAwMC0wMDAwLTAwMDAwMDAwMDAwMCcsIGNvbkxpc3Q9W3t1J3Byb3RvY29sDQpfdmVyc2lv
bic6IDMsIHUnY29ubmVjdGlvbic6IHUnMTkyLjE2OC4xLjIxOi9zcnYvb3ZpcnQnLCB1J3VzZXIn
OiB1J2t2bScsIHUnaWQnOiB1J2RiZWI4YWI0LTg0OWYtNDcyOC04ZWU5LWY4OTFiYjg0Y2UyZid9
XSwgb3B0aW9ucz1Ob25lKQ0KanNvbnJwYy5FeGVjdXRvci82OjpERUJVRzo6MjAxNy0wNy0xNyAx
NDoyNDo0MSwwMDY6OmZpbGVVdGlsczo6MjA5OjpTdG9yYWdlLmZpbGVVdGlsczo6KGNyZWF0ZWRp
cikgQ3JlYXRpbmcgZGlyZWN0b3J5OiAvcmhldi9kYXRhLWNlbnRlci9tbnQvMTkyLjE2OC4xLjIx
Ol9zcnZfb3ZpcnQgbW9kZTogTm9uZQ0KanNvbnJwYy5FeGVjdXRvci82OjpERUJVRzo6MjAxNy0w
Ny0xNyAxNDoyNDo0MSwwMDc6OmZpbGVVdGlsczo6MjE4OjpTdG9yYWdlLmZpbGVVdGlsczo6KGNy
ZWF0ZWRpcikgVXNpbmcgZXhpc3RpbmcgZGlyZWN0b3J5OiAvcmhldi9kYXRhLWNlbnRlci9tbnQv
MTkyLjE2OC4xLjIxOl9zcnZfb3ZpcnQNCmpzb25ycGMuRXhlY3V0b3IvNjo6SU5GTzo6MjAxNy0w
Ny0xNyAxNDoyNDo0MSwwMDc6Om1vdW50OjoyMjY6OnN0b3JhZ2UuTW91bnQ6Oihtb3VudCkgbW91
bnRpbmcgMTkyLjE2OC4xLjIxOi9zcnYvb3ZpcnQgYXQgL3JoZXYvZGF0YS1jZW50ZXIvbW50LzE5
Mi4xNjguMS4yMTpfc3J2X292aXJ0DQpqc29ucnBjLkV4ZWN1dG9yLzY6OkVSUk9SOjoyMDE3LTA3
LTE3IDE0OjI2OjQ2LDA5ODo6aHNtOjoyNDAzOjpTdG9yYWdlLkhTTTo6KGNvbm5lY3RTdG9yYWdl
U2VydmVyKSBDb3VsZCBub3QgY29ubmVjdCB0byBzdG9yYWdlU2VydmVyDQpUcmFjZWJhY2sgKG1v
c3QgcmVjZW50IGNhbGwgbGFzdCk6DQogIEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9zdG9yYWdlL2hz
bS5weSIsIGxpbmUgMjQwMCwgaW4gY29ubmVjdFN0b3JhZ2VTZXJ2ZXINCiAgICBjb25PYmouY29u
bmVjdCgpDQogIEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9zdG9yYWdlL3N0b3JhZ2VTZXJ2ZXIucHki
LCBsaW5lIDQ1NiwgaW4gY29ubmVjdA0KICAgIHJldHVybiBzZWxmLl9tb3VudENvbi5jb25uZWN0
KCkNCiAgRmlsZSAiL3Vzci9zaGFyZS92ZHNtL3N0b3JhZ2Uvc3RvcmFnZVNlcnZlci5weSIsIGxp
bmUgMjM4LCBpbiBjb25uZWN0DQogICAgc2l4LnJlcmFpc2UodCwgdiwgdGIpDQogIEZpbGUgIi91
c3Ivc2hhcmUvdmRzbS9zdG9yYWdlL3N0b3JhZ2VTZXJ2ZXIucHkiLCBsaW5lIDIzMCwgaW4gY29u
bmVjdA0KICAgIHNlbGYuX21vdW50Lm1vdW50KHNlbGYub3B0aW9ucywgc2VsZi5fdmZzVHlwZSwg
Y2dyb3VwPXNlbGYuQ0dST1VQKQ0KICBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNr
YWdlcy92ZHNtL3N0b3JhZ2UvbW91bnQucHkiLCBsaW5lIDIyOSwgaW4gbW91bnQNCiAgICB0aW1l
b3V0PXRpbWVvdXQsIGNncm91cD1jZ3JvdXApDQogIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9z
aXRlLXBhY2thZ2VzL3Zkc20vc3VwZXJ2ZHNtLnB5IiwgbGluZSA1MywgaW4gX19jYWxsX18NCiAg
ICByZXR1cm4gY2FsbE1ldGhvZCgpDQogIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBh
Y2thZ2VzL3Zkc20vc3VwZXJ2ZHNtLnB5IiwgbGluZSA1MSwgaW4gPGxhbWJkYT4NCiAgICAqKmt3
YXJncykNCiAgRmlsZSAiPHN0cmluZz4iLCBsaW5lIDIsIGluIG1vdW50DQogIEZpbGUgIi91c3Iv
bGliNjQvcHl0aG9uMi43L211bHRpcHJvY2Vzc2luZy9tYW5hZ2Vycy5weSIsIGxpbmUgNzczLCBp
biBfY2FsbG1ldGhvZA0KICAgIHJhaXNlIGNvbnZlcnRfdG9fZXJyb3Ioa2luZCwgcmVzdWx0KQ0K
TW91bnRFcnJvcjogKDMyLCAnO21vdW50Lm5mczogQ29ubmVjdGlvbiB0aW1lZCBvdXRcbicpDQoN
Cg0KRnJvbTogPHVzZXJzLWJvdW5jZXNAb3ZpcnQub3JnPiBvbiBiZWhhbGYgb2YgUGhpbGxpcCBC
YWlsZXkgPHBoYmFpbGV5QHJlZGhhdC5jb20+DQpEYXRlOiBUdWVzZGF5LCAxOCBKdWx5IDIwMTcg
YXQgMTM6NDgNClRvOiBMdWNhICdyZW1peF90aicgTG9yZW56ZXR0byA8bG9yZW56ZXR0by5sdWNh
QGdtYWlsLmNvbT4NCkNjOiB1c2VycyA8dXNlcnNAb3ZpcnQub3JnPg0KU3ViamVjdDogUmU6IFtv
dmlydC11c2Vyc10gSG9zdGVkIEVuZ2luZS9ORlMgVHJvdWJsZXMNCg0KT24gTW9uLCBKdWwgMTcs
IDIwMTcgYXQgMzozNCBQTSwgTHVjYSAncmVtaXhfdGonIExvcmVuemV0dG8gPGxvcmVuemV0dG8u
bHVjYUBnbWFpbC5jb208bWFpbHRvOmxvcmVuemV0dG8ubHVjYUBnbWFpbC5jb20+PiB3cm90ZToN
Ck9uIE1vbiwgSnVsIDE3LCAyMDE3IGF0IDk6MDUgUE0sIFBoaWxsaXAgQmFpbGV5IDxwaGJhaWxl
eUByZWRoYXQuY29tPG1haWx0bzpwaGJhaWxleUByZWRoYXQuY29tPj4gd3JvdGU6DQo+IEhpLA0K
Pg0KPiBJJ20gaGF2aW5nIHRyb3VibGUgd2l0aCBteSBob3N0ZWQgZW5naW5lIHNldHVwICh2NC4w
KSBhbmQgY291bGQgdXNlIHNvbWUNCj4gaGVscC4gVGhlIHByb2JsZW0gSSdtIGhhdmluZyBpcyB0
aGF0IHdoZW5ldmVyIEkgdHJ5IHRvIGFkZCBhZGRpdGlvbmFsIGhvc3RzDQo+IHRvIHRoZSBzZXR1
cCB2aWEgd2ViYWRtaW4sIHRoZSBvcGVyYXRpb24gZmFpbHMgZHVlIHRvIHN0b3JhZ2UtcmVsYXRl
ZA0KPiBpc3N1ZXMuDQo+DQo+IHdlYmFkbWluIHNob3dzIHRoZSBmb2xsb3dpbmcgZXJyb3IgbWVz
c2FnZXM6DQo+DQo+ICJIb3N0IDxob3N0IG5hbWU+IGNhbm5vdCBhY2Nlc3MgdGhlIFN0b3JhZ2Ug
RG9tYWluKHMpIGhvc3RlZF9zdG9yYWdlDQo+IGF0dGFjaGVkIHRvIHRoZSBEYXRhIENlbnRlciBE
ZWZhdWx0LiBTZXR0aW5nIEhvc3Qgc3RhdGUgdG8gTm9uLU9wZXJhdGlvbmFsLg0KPiBGYWlsZWQg
dG8gY29ubmVjdCBIb3N0IG92aXJ0LW5vZGUtMSB0byBTdG9yYWdlIFBvb2wgRGVmYXVsdCINCj4N
Cg0KSGkgUGhpbGxpcCwNCg0KeW91ciBob3N0ZWQgZW5naW5lIHN0b3JhZ2UgaXMgb24gbmZzLCBy
aWdodD8gRGlkIHlvdSB0ZXN0IGlmIHlvdSBjYW4NCm1vdW50IG1hbnVhbGx5IG9uIGVhY2ggaG9z
dD8NCkhpIEx1Y2EsDQoNClllcywgYm90aCBzdG9yYWdlIGRvbWFpbnMgYXJlIG9uIE5GUyAodjMp
IGFuZCBJIGFtIGFibGUgdG8gc3VjY2Vzc2Z1bGx5IG1vdW50IHRoZW0gbWFudWFsbHkgb24gdGhl
IGhvc3RzLg0KDQpMdWNhDQoNCg0KDQotLQ0KIkUnIGFzc3VyZG8gaW1waWVnYXJlIGdsaSB1b21p
bmkgZGkgaW50ZWxsaWdlbnphIGVjY2VsbGVudGUgcGVyIGZhcmUNCmNhbGNvbGkgY2hlIHBvdHJl
YmJlcm8gZXNzZXJlIGFmZmlkYXRpIGEgY2hpdW5xdWUgc2Ugc2kgdXNhc3Nlcm8gZGVsbGUNCm1h
Y2NoaW5lIg0KR290dGZyaWVkIFdpbGhlbG0gdm9uIExlaWJuaXR6LCBGaWxvc29mbyBlIE1hdGVt
YXRpY28gKDE2NDYtMTcxNikNCg0KIkludGVybmV0IMOoIGxhIHBpw7kgZ3JhbmRlIGJpYmxpb3Rl
Y2EgZGVsIG1vbmRvLg0KTWEgaWwgcHJvYmxlbWEgw6ggY2hlIGkgbGlicmkgc29ubyB0dXR0aSBz
cGFyc2kgc3VsIHBhdmltZW50byINCkpvaG4gQWxsZW4gUGF1bG9zLCBNYXRlbWF0aWNvICgxOTQ1
LXZpdmVudGUpDQoNCkx1Y2EgJ3JlbWl4X3RqJyBMb3JlbnpldHRvLCBodHRwOi8vd3d3LnJlbWl4
dGoubmV0ICwgPGxvcmVuemV0dG8ubHVjYUBnbWFpbC5jb208bWFpbHRvOmxvcmVuemV0dG8ubHVj
YUBnbWFpbC5jb20+Pg0KDQo=
--_000_25E6C4DA741949D8A77A14C928CF7722acroniscom_
Content-Type: text/html; charset="utf-8"
Content-ID: <5C498FBBA2A36F47B6957EB3F4386884(a)acronis.com>
Content-Transfer-Encoding: base64
PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4
bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo
dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo
dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp
dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l
dGEgbmFtZT0iVGl0bGUiIGNvbnRlbnQ9IiI+DQo8bWV0YSBuYW1lPSJLZXl3b3JkcyIgY29udGVu
dD0iIj4NCjxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg
KGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxlPjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8N
CkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0
IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1mYWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJ
cGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8N
CnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBjbTsN
CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjEyLjBwdDsNCglmb250LWZhbWls
eToiVGltZXMgTmV3IFJvbWFuIjt9DQphOmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1z
dHlsZS1wcmlvcml0eTo5OTsNCgljb2xvcjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxp
bmU7fQ0KYTp2aXNpdGVkLCBzcGFuLk1zb0h5cGVybGlua0ZvbGxvd2VkDQoJe21zby1zdHlsZS1w
cmlvcml0eTo5OTsNCgljb2xvcjpwdXJwbGU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9
DQpzcGFuLmhvZW56Yg0KCXttc28tc3R5bGUtbmFtZTpob2VuemI7fQ0Kc3Bhbi5FbWFpbFN0eWxl
MTgNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7DQoJZm9udC1mYW1pbHk6Q2FsaWJy
aTsNCgljb2xvcjp3aW5kb3d0ZXh0O30NCnNwYW4ubXNvSW5zDQoJe21zby1zdHlsZS10eXBlOmV4
cG9ydC1vbmx5Ow0KCW1zby1zdHlsZS1uYW1lOiIiOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxp
bmU7DQoJY29sb3I6dGVhbDt9DQouTXNvQ2hwRGVmYXVsdA0KCXttc28tc3R5bGUtdHlwZTpleHBv
cnQtb25seTsNCglmb250LXNpemU6MTAuMHB0O30NCkBwYWdlIFdvcmRTZWN0aW9uMQ0KCXtzaXpl
OjU5NS4wcHQgODQyLjBwdDsNCgltYXJnaW46Mi4wY20gNDIuNXB0IDIuMGNtIDMuMGNtO30NCmRp
di5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT4NCjwvaGVh
ZD4NCjxib2R5IGJnY29sb3I9IndoaXRlIiBsYW5nPSJFTi1HQiIgbGluaz0iYmx1ZSIgdmxpbms9
InB1cnBsZSI+DQo8ZGl2IGNsYXNzPSJXb3JkU2VjdGlvbjEiPg0KPHAgY2xhc3M9Ik1zb05vcm1h
bCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28t
ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+UGhpbGxpcCw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh
bWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpw
Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl
OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5U
aGUgcmVsZXZhbnQgbGluZXMgZnJvbSB0aGUgdmRzbSBsb2dzIGFyZSB0aGUgZm9sbG93aW5nOjxv
OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm
b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6
RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZh
cmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPmpzb25ycGMuRXhlY3V0b3IvNjo6SU5GTzo6MjAxNy0wNy0x
NyAxNDoyNDo0MSwwMDU6OmxvZ1V0aWxzOjo0OTo6ZGlzcGF0Y2hlcjo6KHdyYXBwZXIpIFJ1biBh
bmQgcHJvdGVjdDogY29ubmVjdFN0b3JhZ2VTZXJ2ZXIoZG9tVHlwZT0xLCBzcFVVSUQ9dScwMDAw
MDAwMC0wMDAwLTAwMDAtMDAwMC0wMDAwMDAwMDAwMDAnLA0KIGNvbkxpc3Q9W3t1J3Byb3RvY29s
PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9
ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+X3ZlcnNpb24nOiAzLCB1J2Nvbm5lY3Rpb24nOiB1JzE5Mi4xNjguMS4yMTovc3J2
L292aXJ0JywgdSd1c2VyJzogdSdrdm0nLCB1J2lkJzogdSdkYmViOGFiNC04NDlmLTQ3MjgtOGVl
OS1mODkxYmI4NGNlMmYnfV0sIG9wdGlvbnM9Tm9uZSk8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh
bWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5qc29ucnBjLkV4ZWN1dG9y
LzY6OkRFQlVHOjoyMDE3LTA3LTE3IDE0OjI0OjQxLDAwNjo6ZmlsZVV0aWxzOjoyMDk6OlN0b3Jh
Z2UuZmlsZVV0aWxzOjooY3JlYXRlZGlyKSBDcmVhdGluZyBkaXJlY3Rvcnk6IC9yaGV2L2RhdGEt
Y2VudGVyL21udC8xOTIuMTY4LjEuMjE6X3Nydl9vdmlydA0KIG1vZGU6IE5vbmU8bzpwPjwvbzpw
Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl
OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5q
c29ucnBjLkV4ZWN1dG9yLzY6OkRFQlVHOjoyMDE3LTA3LTE3IDE0OjI0OjQxLDAwNzo6ZmlsZVV0
aWxzOjoyMTg6OlN0b3JhZ2UuZmlsZVV0aWxzOjooY3JlYXRlZGlyKSBVc2luZyBleGlzdGluZyBk
aXJlY3Rvcnk6IC9yaGV2L2RhdGEtY2VudGVyL21udC8xOTIuMTY4LjEuMjE6X3Nydl9vdmlydDxv
OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm
b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6
RU4tVVMiPmpzb25ycGMuRXhlY3V0b3IvNjo6SU5GTzo6MjAxNy0wNy0xNyAxNDoyNDo0MSwwMDc6
Om1vdW50OjoyMjY6OnN0b3JhZ2UuTW91bnQ6Oihtb3VudCkgbW91bnRpbmcgMTkyLjE2OC4xLjIx
Oi9zcnYvb3ZpcnQgYXQgL3JoZXYvZGF0YS1jZW50ZXIvbW50LzE5Mi4xNjguMS4yMTpfc3J2X292
aXJ0PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5n
dWFnZTpFTi1VUyI+anNvbnJwYy5FeGVjdXRvci82OjpFUlJPUjo6MjAxNy0wNy0xNyAxNDoyNjo0
NiwwOTg6OmhzbTo6MjQwMzo6U3RvcmFnZS5IU006Oihjb25uZWN0U3RvcmFnZVNlcnZlcikgQ291
bGQgbm90IGNvbm5lY3QgdG8gc3RvcmFnZVNlcnZlcjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt
aWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPlRyYWNlYmFjayAobW9zdCBy
ZWNlbnQgY2FsbCBsYXN0KTo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9y
bWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21z
by1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgRmlsZSAmcXVvdDsvdXNyL3NoYXJlL3Zk
c20vc3RvcmFnZS9oc20ucHkmcXVvdDssIGxpbmUgMjQwMCwgaW4gY29ubmVjdFN0b3JhZ2VTZXJ2
ZXI8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls
ZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1
YWdlOkVOLVVTIj4mbmJzcDsgJm5ic3A7IGNvbk9iai5jb25uZWN0KCk8bzpwPjwvbzpwPjwvc3Bh
bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBw
dDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsg
RmlsZSAmcXVvdDsvdXNyL3NoYXJlL3Zkc20vc3RvcmFnZS9zdG9yYWdlU2VydmVyLnB5JnF1b3Q7
LCBsaW5lIDQ1NiwgaW4gY29ubmVjdDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN
c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGli
cmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOyAmbmJzcDsgcmV0dXJuIHNlbGYu
X21vdW50Q29uLmNvbm5lY3QoKTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7
bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOyBGaWxlICZxdW90Oy91c3Ivc2hhcmUv
dmRzbS9zdG9yYWdlL3N0b3JhZ2VTZXJ2ZXIucHkmcXVvdDssIGxpbmUgMjM4LCBpbiBjb25uZWN0
PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9
ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+Jm5ic3A7ICZuYnNwOyBzaXgucmVyYWlzZSh0LCB2LCB0Yik8bzpwPjwvbzpwPjwv
c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEx
LjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJz
cDsgRmlsZSAmcXVvdDsvdXNyL3NoYXJlL3Zkc20vc3RvcmFnZS9zdG9yYWdlU2VydmVyLnB5JnF1
b3Q7LCBsaW5lIDIzMCwgaW4gY29ubmVjdDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNh
bGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOyAmbmJzcDsgc2VsZi5fbW91
bnQubW91bnQoc2VsZi5vcHRpb25zLCBzZWxmLl92ZnNUeXBlLCBjZ3JvdXA9c2VsZi5DR1JPVVAp
PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9
ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+Jm5ic3A7IEZpbGUgJnF1b3Q7L3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2Fn
ZXMvdmRzbS9zdG9yYWdlL21vdW50LnB5JnF1b3Q7LCBsaW5lIDIyOSwgaW4gbW91bnQ8bzpwPjwv
bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1z
aXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT
Ij4mbmJzcDsgJm5ic3A7IHRpbWVvdXQ9dGltZW91dCwgY2dyb3VwPWNncm91cCk8bzpwPjwvbzpw
Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXpl
OjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4m
bmJzcDsgRmlsZSAmcXVvdDsvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNtL3N1
cGVydmRzbS5weSZxdW90OywgbGluZSA1MywgaW4gX19jYWxsX188bzpwPjwvbzpwPjwvc3Bhbj48
L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtm
b250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgJm5i
c3A7IHJldHVybiBjYWxsTWV0aG9kKCk8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxp
YnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgRmlsZSAmcXVvdDsvdXNyL2xp
Yi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNtL3N1cGVydmRzbS5weSZxdW90OywgbGluZSA1
MSwgaW4gJmx0O2xhbWJkYSZndDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJp
O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDsgJm5ic3A7ICoqa3dhcmdzKTxvOnA+
PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250
LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t
VVMiPiZuYnNwOyBGaWxlICZxdW90OyZsdDtzdHJpbmcmZ3Q7JnF1b3Q7LCBsaW5lIDIsIGluIG1v
dW50PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5n
dWFnZTpFTi1VUyI+Jm5ic3A7IEZpbGUgJnF1b3Q7L3Vzci9saWI2NC9weXRob24yLjcvbXVsdGlw
cm9jZXNzaW5nL21hbmFnZXJzLnB5JnF1b3Q7LCBsaW5lIDc3MywgaW4gX2NhbGxtZXRob2Q8bzpw
PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9u
dC1zaXplOjExLjBwdDtmb250LWZhbWlseTpDYWxpYnJpO21zby1mYXJlYXN0LWxhbmd1YWdlOkVO
LVVTIj4mbmJzcDsgJm5ic3A7IHJhaXNlIGNvbnZlcnRfdG9fZXJyb3Ioa2luZCwgcmVzdWx0KTxv
OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJm
b250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OkNhbGlicmk7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6
RU4tVVMiPk1vdW50RXJyb3I6ICgzMiwgJzttb3VudC5uZnM6DQo8Yj5Db25uZWN0aW9uIHRpbWVk
IG91dDwvYj5cbicpPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6Q2FsaWJyaTttc28tZmFy
ZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xh
c3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6
Q2FsaWJyaTttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3Nw
YW4+PC9wPg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjQjVDNERG
IDEuMHB0O3BhZGRpbmc6My4wcHQgMGNtIDBjbSAwY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PGI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OkNhbGlicmk7Y29sb3I6YmxhY2siPkZyb206IDwv
c3Bhbj4NCjwvYj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6Q2FsaWJyaTtjb2xvcjpibGFjayI+
Jmx0O3VzZXJzLWJvdW5jZXNAb3ZpcnQub3JnJmd0OyBvbiBiZWhhbGYgb2YgUGhpbGxpcCBCYWls
ZXkgJmx0O3BoYmFpbGV5QHJlZGhhdC5jb20mZ3Q7PGJyPg0KPGI+RGF0ZTogPC9iPlR1ZXNkYXks
IDE4IEp1bHkgMjAxNyBhdCAxMzo0ODxicj4NCjxiPlRvOiA8L2I+THVjYSAncmVtaXhfdGonIExv
cmVuemV0dG8gJmx0O2xvcmVuemV0dG8ubHVjYUBnbWFpbC5jb20mZ3Q7PGJyPg0KPGI+Q2M6IDwv
Yj51c2VycyAmbHQ7dXNlcnNAb3ZpcnQub3JnJmd0Ozxicj4NCjxiPlN1YmplY3Q6IDwvYj5SZTog
W292aXJ0LXVzZXJzXSBIb3N0ZWQgRW5naW5lL05GUyBUcm91YmxlczxvOnA+PC9vOnA+PC9zcGFu
PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9v
OnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPGRpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs
Ij5PbiBNb24sIEp1bCAxNywgMjAxNyBhdCAzOjM0IFBNLCBMdWNhICdyZW1peF90aicgTG9yZW56
ZXR0byAmbHQ7PGEgaHJlZj0ibWFpbHRvOmxvcmVuemV0dG8ubHVjYUBnbWFpbC5jb20iIHRhcmdl
dD0iX2JsYW5rIj5sb3JlbnpldHRvLmx1Y2FAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PG86cD48
L286cD48L3A+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29s
aWQgI0NDQ0NDQyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQu
OHB0O21hcmdpbi1yaWdodDowY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdp
bi1ib3R0b206MTIuMHB0Ij5PbiBNb24sIEp1bCAxNywgMjAxNyBhdCA5OjA1IFBNLCBQaGlsbGlw
IEJhaWxleSAmbHQ7PGEgaHJlZj0ibWFpbHRvOnBoYmFpbGV5QHJlZGhhdC5jb20iPnBoYmFpbGV5
QHJlZGhhdC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+DQomZ3Q7IEhpLDxicj4NCiZndDs8YnI+DQom
Z3Q7IEknbSBoYXZpbmcgdHJvdWJsZSB3aXRoIG15IGhvc3RlZCBlbmdpbmUgc2V0dXAgKHY0LjAp
IGFuZCBjb3VsZCB1c2Ugc29tZTxicj4NCiZndDsgaGVscC4gVGhlIHByb2JsZW0gSSdtIGhhdmlu
ZyBpcyB0aGF0IHdoZW5ldmVyIEkgdHJ5IHRvIGFkZCBhZGRpdGlvbmFsIGhvc3RzPGJyPg0KJmd0
OyB0byB0aGUgc2V0dXAgdmlhIHdlYmFkbWluLCB0aGUgb3BlcmF0aW9uIGZhaWxzIGR1ZSB0byBz
dG9yYWdlLXJlbGF0ZWQ8YnI+DQomZ3Q7IGlzc3Vlcy48YnI+DQomZ3Q7PGJyPg0KJmd0OyB3ZWJh
ZG1pbiBzaG93cyB0aGUgZm9sbG93aW5nIGVycm9yIG1lc3NhZ2VzOjxicj4NCiZndDs8YnI+DQom
Z3Q7ICZxdW90O0hvc3QgJmx0O2hvc3QgbmFtZSZndDsgY2Fubm90IGFjY2VzcyB0aGUgU3RvcmFn
ZSBEb21haW4ocykgaG9zdGVkX3N0b3JhZ2U8YnI+DQomZ3Q7IGF0dGFjaGVkIHRvIHRoZSBEYXRh
IENlbnRlciBEZWZhdWx0LiBTZXR0aW5nIEhvc3Qgc3RhdGUgdG8gTm9uLU9wZXJhdGlvbmFsLjxi
cj4NCiZndDsgRmFpbGVkIHRvIGNvbm5lY3QgSG9zdCBvdmlydC1ub2RlLTEgdG8gU3RvcmFnZSBQ
b29sIERlZmF1bHQmcXVvdDs8YnI+DQomZ3Q7PGJyPg0KPGJyPg0KSGkgUGhpbGxpcCw8YnI+DQo8
YnI+DQp5b3VyIGhvc3RlZCBlbmdpbmUgc3RvcmFnZSBpcyBvbiBuZnMsIHJpZ2h0PyBEaWQgeW91
IHRlc3QgaWYgeW91IGNhbjxicj4NCm1vdW50IG1hbnVhbGx5IG9uIGVhY2ggaG9zdD88bzpwPjwv
bzpwPjwvcD4NCjwvYmxvY2txdW90ZT4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5IaSBM
dWNhLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs
Ij5ZZXMsIGJvdGggc3RvcmFnZSBkb21haW5zIGFyZSBvbiBORlMgKHYzKSBhbmQgSSBhbSBhYmxl
IHRvIHN1Y2Nlc3NmdWxseSBtb3VudCB0aGVtIG1hbnVhbGx5IG9uIHRoZSBob3N0cy48bzpwPjwv
bzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7
PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVy
LWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdp
bi1sZWZ0OjQuOHB0O21hcmdpbi1yaWdodDowY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw
YW4gY2xhc3M9ImhvZW56YiI+PHNwYW4gc3R5bGU9ImNvbG9yOiM4ODg4ODgiPkx1Y2E8L3NwYW4+
PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjojODg4ODg4Ij48YnI+DQo8YnI+DQo8YnI+DQo8YnI+
DQo8c3BhbiBjbGFzcz0iaG9lbnpiIj4tLTwvc3Bhbj48YnI+DQo8c3BhbiBjbGFzcz0iaG9lbnpi
Ij4mcXVvdDtFJyBhc3N1cmRvIGltcGllZ2FyZSBnbGkgdW9taW5pIGRpIGludGVsbGlnZW56YSBl
Y2NlbGxlbnRlIHBlciBmYXJlPC9zcGFuPjxicj4NCjxzcGFuIGNsYXNzPSJob2VuemIiPmNhbGNv
bGkgY2hlIHBvdHJlYmJlcm8gZXNzZXJlIGFmZmlkYXRpIGEgY2hpdW5xdWUgc2Ugc2kgdXNhc3Nl
cm8gZGVsbGU8L3NwYW4+PGJyPg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+bWFjY2hpbmUmcXVvdDs8
L3NwYW4+PGJyPg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+R290dGZyaWVkIFdpbGhlbG0gdm9uIExl
aWJuaXR6LCBGaWxvc29mbyBlIE1hdGVtYXRpY28gKDE2NDYtMTcxNik8L3NwYW4+PGJyPg0KPGJy
Pg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+JnF1b3Q7SW50ZXJuZXQgw6ggbGEgcGnDuSBncmFuZGUg
YmlibGlvdGVjYSBkZWwgbW9uZG8uPC9zcGFuPjxicj4NCjxzcGFuIGNsYXNzPSJob2VuemIiPk1h
IGlsIHByb2JsZW1hIMOoIGNoZSBpIGxpYnJpIHNvbm8gdHV0dGkgc3BhcnNpIHN1bCBwYXZpbWVu
dG8mcXVvdDs8L3NwYW4+PGJyPg0KPHNwYW4gY2xhc3M9ImhvZW56YiI+Sm9obiBBbGxlbiBQYXVs
b3MsIE1hdGVtYXRpY28gKDE5NDUtdml2ZW50ZSk8L3NwYW4+PGJyPg0KPGJyPg0KPHNwYW4gY2xh
c3M9ImhvZW56YiI+THVjYSAncmVtaXhfdGonIExvcmVuemV0dG8sIDxhIGhyZWY9Imh0dHA6Ly93
d3cucmVtaXh0ai5uZXQiIHRhcmdldD0iX2JsYW5rIj4NCmh0dHA6Ly93d3cucmVtaXh0ai5uZXQ8
L2E+ICwgJmx0OzxhIGhyZWY9Im1haWx0bzpsb3JlbnpldHRvLmx1Y2FAZ21haWwuY29tIj5sb3Jl
bnpldHRvLmx1Y2FAZ21haWwuY29tPC9hPiZndDs8L3NwYW4+PC9zcGFuPjxvOnA+PC9vOnA+PC9w
Pg0KPC9ibG9ja3F1b3RlPg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNw
OzwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg==
--_000_25E6C4DA741949D8A77A14C928CF7722acroniscom_--
11:04 a.m.
On Tue, Jul 18, 2017 at 7:09 AM, Pavel Gashev <Pax(a)acronis.com> wrote:
Phillip,
The relevant lines from the vdsm logs are the following:
jsonrpc.Executor/6::INFO::2017-07-17 14:24:41,005::logUtils::49::dispatcher::(wrapper)
Run and protect: connectStorageServer(domType=1,
spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'protocol
_version': 3, u'connection': u'192.168.1.21:/srv/ovirt',
u'user': u'kvm',
u'id': u'dbeb8ab4-849f-4728-8ee9-f891bb84ce2f'}], options=None)
jsonrpc.Executor/6::DEBUG::2017-07-17
14:24:41,006::fileUtils::209::Storage.fileUtils::(createdir)
Creating directory: /rhev/data-center/mnt/192.168.1.21:_srv_ovirt mode:
None
jsonrpc.Executor/6::DEBUG::2017-07-17
14:24:41,007::fileUtils::218::Storage.fileUtils::(createdir)
Using existing directory: /rhev/data-center/mnt/192.168.1.21:_srv_ovirt
jsonrpc.Executor/6::INFO::2017-07-17 14:24:41,007::mount::226::storage.Mount::(mount)
mounting 192.168.1.21:/srv/ovirt at /rhev/data-center/mnt/192.168.
1.21:_srv_ovirt
jsonrpc.Executor/6::ERROR::2017-07-17 14:26:46,098::hsm::2403::
Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
File "/usr/share/vdsm/storage/hsm.py", line 2400, in
connectStorageServer
conObj.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 456, in connect
return self._mountCon.connect()
File "/usr/share/vdsm/storage/storageServer.py", line 238, in connect
six.reraise(t, v, tb)
File "/usr/share/vdsm/storage/storageServer.py", line 230, in connect
self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)
File "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line
229, in mount
timeout=timeout, cgroup=cgroup)
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in
__call__
return callMethod()
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in
<lambda>
**kwargs)
File "<string>", line 2, in mount
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
MountError: (32, ';mount.nfs: *Connection timed out*\n')
I saw this as well, but I don't understand why I don't have the same
problem when mounting the domain manually and writing to it as the vdsm
user. Thanks for pointing it out, though. I should have included that in
the original message.
*From: *<users-bounces(a)ovirt.org> on behalf of Phillip Bailey <
phbailey(a)redhat.com>
*Date: *Tuesday, 18 July 2017 at 13:48
*To: *Luca 'remix_tj' Lorenzetto <lorenzetto.luca(a)gmail.com>
*Cc: *users <users(a)ovirt.org>
*Subject: *Re: [ovirt-users] Hosted Engine/NFS Troubles
On Mon, Jul 17, 2017 at 3:34 PM, Luca 'remix_tj' Lorenzetto <
lorenzetto.luca(a)gmail.com> wrote:
On Mon, Jul 17, 2017 at 9:05 PM, Phillip Bailey <phbailey(a)redhat.com>
wrote:
> Hi,
>
> I'm having trouble with my hosted engine setup (v4.0) and could use some
> help. The problem I'm having is that whenever I try to add additional
hosts
> to the setup via webadmin, the operation fails due to storage-related
> issues.
>
> webadmin shows the following error messages:
>
> "Host <host name> cannot access the Storage Domain(s) hosted_storage
> attached to the Data Center Default. Setting Host state to
Non-Operational.
> Failed to connect Host ovirt-node-1 to Storage Pool Default"
>
Hi Phillip,
your hosted engine storage is on nfs, right? Did you test if you can
mount manually on each host?
Hi Luca,
Yes, both storage domains are on NFS (v3) and I am able to successfully
mount them manually on the hosts.
Luca
--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <
lorenzetto.luca(a)gmail.com>
7:04 a.m.
--_000_150037588204129266leedsbeckettacuk_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
I used the troubleshooting guide at
documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/
and exported using
(rw,sync,no_subtree_check,all_squash,anonuid=3D36,anongid=3D36)
This was for the DATA_DOMAIN not hosted storage but it may help.
Regards,
Paul S.
________________________________
From: users-bounces(a)ovirt.org <users-bounces(a)ovirt.org> on behalf of Philli=
p Bailey <phbailey(a)redhat.com Sent: 17 July 2017 20:05
To: users
Subject: [ovirt-users] Hosted Engine/NFS Troubles
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some he=
lp. The problem I'm having is that whenever I try to add additional hosts t=
o the setup via webadmin, the operation fails due to storage-related issues=
.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage attach=
ed to the Data Center Default. Setting Host state to Non-Operational.
Failed to connect Host ovirt-node-1 to Storage Pool Default"
The VDSM log from the host shows the following error message:
"Thread-18::ERROR::2017-07-17 13:01:11,483::sdc::146::Storage.StorageDomain=
Cache::(_findDomain) domain ca044720-e5cf-40a8-8b21-57a17026db7c not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'ca044720-e5cf-=
40a8-8b21-57a17026db7c',)"
The engine log shows the following error messages:
"2017-07-17 18:32:11,409 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.I=
rsProxyData] (org.ovirt.thread.pool-6-thread-34) [] Domain 'ca044720-e5cf-4=
0a8-8b21-57a17026db7c:hosted_storage' was reported with error code '358'
2017-07-17 18:32:11,410 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand=
] (org.ovirt.thread.pool-6-thread-34) [] Storage Domain 'hosted_storage' of=
pool 'Default' is in problem in host 'ovirt-node-1'
2017-07-17 18:32:11,487 ERROR [org.ovirt.engine.core.dal.dbbroker.auditlogh=
andling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34) [] Correlatio=
n ID: null, Call Stack: null, Custom Event ID: -1, Message: Host ovirt-node=
-1 reports about one of the Active Storage Domains as Problematic."
I have ownership set to vdsm/kvm and full rwx rights enabled on both direct=
ories. I have successfully mounted both the master domain and the hosted_st=
orage manually on one of the hosts I'm trying to add. I have attached the e=
ngine log and the VDSM log for that host.
Could someone please help me figure out what's causing this?
-Phillip Bailey
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
--_000_150037588204129266leedsbeckettacuk_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none"><!--P{margin-top:0;margin-b=
ottom:0;} --></style </head <body dir=3D"ltr"
style=3D"font-size:12pt;color:#000000;background-color:#F=
FFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"
<p>I used the troubleshooting guide at </p
<p><br </p
<p>documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/=
</p <p><br
</p <p>and exported
using</p <p><br </p
<p>(rw,sync,no_subtree_check,all_squash,anonuid=3D36,anongid=3D36) </p <p><br
</p <p>This was for the
DATA_DOMAIN not hosted storage but it may help.</p
<p><br </p
<p>Regards,</p <p><br </p <p>Paul S.<br </p <div style=3D"color: rgb(33,
33, 33);" <hr tabindex=3D"-1"
style=3D"display:inline-block; width:98%" <div
id=3D"divRplyFwdMsg" dir=3D"ltr"><font
style=3D"font-size:11pt" face=
=3D"Calibri, sans-serif" color=3D"#000000"><b>From:</b>
users-bounces@ovirt=
.org &lt;users-bounces(a)ovirt.org&gt; on behalf of Phillip Bailey <phbail=
ey(a)redhat.com&gt;<br <b>Sent:</b> 17 July
2017 20:05<br <b>To:</b>
users<br <b>Subject:</b>
[ovirt-users] Hosted Engine/NFS Troubles</font
<div> </div
</div <div <div
dir=3D"ltr" <div><span
style=3D"font-size:12.8px">Hi,</span <div
style=3D"font-size:12.8px"><br
</div <div
style=3D"font-size:12.8px">I'm having trouble with my hosted engine se=
tup (v4.0) and could use some help. <span
style=3D"font-size:12.8px">T=
he problem I'm having is that whenever I try to add additional hosts to the=
setup via webadmin, the operation fails
due to storage-related issues. </span></div <div style=3D"font-size:12.8px"><br webadmin shows the following error messages:</div <div style=3D"font-size:12.8px"><br </div <div
style=3D"font-size:12.8px">"Host <host name> cannot
access =
the Storage Domain(s) hosted_storage attached to the Data Center Default. S=
etting Host state to Non-Operational.</div <div
style=3D"font-size:12.8px"><span
style=3D"font-size:12.8px">Failed to =
connect Host ovirt-node-1 to Storage Pool Default"<br </span><br
</div <div
style=3D"font-size:12.8px"><br
</div <div
style=3D"font-size:12.8px">The VDSM log from the host shows the follow=
ing error message:</div <div
style=3D"font-size:12.8px"><br
</div <div
style=3D"font-size:12.8px">"<span
style=3D"font-size:12.8px">Thre=
ad-18::ERROR::2017-07-17 13:01:11,483::sdc::146::<wbr>Storage.StorageDomain=
Cache::(_<wbr>findDomain) domain ca044720-e5cf-40a8-8b21-<wbr>57a17026db7c =
not found</span></div <div
style=3D"font-size:12.8px">Traceback (most recent call last):</div <div style=3D"font-size:12.8px"> File
"/usr/share/vdsm/storage/s=
dc.<wbr>py", line 144, in _findDomain</div <div
style=3D"font-size:12.8px"> dom =3D
findMethod(sdUUID)</d=
iv <div style=3D"font-size:12.8px"> File
"/usr/share/vdsm/storage/s=
dc.<wbr>py", line 174, in _findUnfetchedDomain</div <div style=3D"font-size:12.8px">
raise se.StorageDomainDoesNot=
Exist(<wbr>sdUUID)</div <div
style=3D"font-size:12.8px">StorageDomainDoesNotExist: Storage domain d=
oes not exist:
(u'ca044720-e5cf-40a8-8b21-<wbr>57a17026db7c',)"</div <div style=3D"font-size:12.8px"><br <br </div <div style=3D"font-size:12.8px">The engine log
shows the following error me=
ssages:<br <br <div
style=3D"font-size:12.8px">"2017-07-17 18:32:11,409 ERROR [org.ov=
irt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>IrsProxyData] (org.ovirt.thre=
ad.pool-6-<wbr>thread-34) [] Domain
'ca044720-e5cf-40a8-8b21-<wbr>57a17026d=
b7c:hosted_storage' was reported with error
code '358'</div <div
style=3D"font-size:12.8px">2017-07-17 18:32:11,410 ERROR [org.ovirt.en=
gine.core.bll.<wbr>InitVdsOnUpCommand] (org.ovirt.thread.pool-6-<wbr>thread=
-34) [] Storage Domain 'hosted_storage' of pool 'Default' is in problem in
=
host 'ovirt-node-1'</div <div
style=3D"font-size:12.8px">2017-07-17 18:32:11,487 ERROR [org.ovirt.en=
gine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (org.ov=
irt.thread.pool-6-<wbr>thread-34) [] Correlation ID: null, Call Stack: null=
, Custom Event ID: -1, Message: Host
ovirt-node-1 reports about one of the Active Storage Domains as Problemati=
c."</div </div <div style=3D"font-size:12.8px"><br </div <div
style=3D"font-size:12.8px"><br
</div <div
style=3D"font-size:12.8px">I have ownership set to vdsm/kvm and full r=
wx rights enabled on both directories. I have successfully mounted both the=
master domain and the hosted_storage manually on one of the hosts I'm tryi=
ng to add. I have attached the engine
log and the VDSM log for that host.<br
<br Could someone please help me figure
out what's causing this?</div
</div <div><br </div
<div <div
class=3D"gmail_signature" <div
dir=3D"ltr">-Phillip Bailey</div
</div </div </div
</div </div To view the terms under which this email is distributed, please
go to:- <br=
<a
href=3D"http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html"=
target=3D"_blank">http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaim=
er.html</a <p></p </body
</html
--_000_150037588204129266leedsbeckettacuk_--
10:58 a.m.
On Tue, Jul 18, 2017 at 7:04 AM, Staniforth, Paul <
P.Staniforth(a)leedsbeckett.ac.uk> wrote:
I used the troubleshooting guide at
documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/
and exported using
(rw,sync,no_subtree_check,all_squash,anonuid=36,anongid=36)
This was for the DATA_DOMAIN not hosted storage but it may help.
My exports file has the same line, so I don't believe that's the problem.
I
went through the recommended steps on the troubleshooting guide and
everything looks to be fine, but the problem still exists. The
troubleshooting guide never came up in my previous googling of the problem
though, so thanks for pointing me to that. =)
Regards,
Paul S.
------------------------------
*From:* users-bounces(a)ovirt.org <users-bounces(a)ovirt.org> on behalf of
Phillip Bailey <phbailey(a)redhat.com>
*Sent:* 17 July 2017 20:05
*To:* users
*Subject:* [ovirt-users] Hosted Engine/NFS Troubles
Hi,
I'm having trouble with my hosted engine setup (v4.0) and could use some
help. The problem I'm having is that whenever I try to add additional
hosts to the setup via webadmin, the operation fails due to storage-related
issues.
webadmin shows the following error messages:
"Host <host name> cannot access the Storage Domain(s) hosted_storage
attached to the Data Center Default. Setting Host state to Non-Operational.
Failed to connect Host ovirt-node-1 to Storage Pool Default"
The VDSM log from the host shows the following error message:
"Thread-18::ERROR::2017-07-17 13:01:11,483::sdc::146::Storag
e.StorageDomainCache::(_findDomain) domain ca044720-e5cf-40a8-8b21-57a17026db7c
not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 144, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 174, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'ca044720-e5cf-40a8-8b21-57a17026db7c',)"
The engine log shows the following error messages:
"2017-07-17 18:32:11,409 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(org.ovirt.thread.pool-6-thread-34) [] Domain
'ca044720-e5cf-40a8-8b21-57a17026db7c:hosted_storage' was reported with
error code '358'
2017-07-17 18:32:11,410 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(org.ovirt.thread.pool-6-thread-34) [] Storage Domain 'hosted_storage' of
pool 'Default' is in problem in host 'ovirt-node-1'
2017-07-17 18:32:11,487 ERROR [org.ovirt.engine.core.dal.dbb
roker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-34)
[] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message:
Host ovirt-node-1 reports about one of the Active Storage Domains as
Problematic."
I have ownership set to vdsm/kvm and full rwx rights enabled on both
directories. I have successfully mounted both the master domain and the
hosted_storage manually on one of the hosts I'm trying to add. I have
attached the engine log and the VDSM log for that host.
Could someone please help me figure out what's causing this?
-Phillip Bailey
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
2492
days inactive
2494
days old
6 comments
4 participants
participants (4)
-
Luca 'remix_tj' Lorenzetto -
Pavel Gashev -
Phillip Bailey -
Staniforth, Paul