From fromani at redhat.com Thu Dec 14 12:22:36 2017 Content-Type: multipart/mixed; boundary="===============6930952422421919488==" MIME-Version: 1.0 From: Francesco Romani To: devel at ovirt.org Subject: Re: [ovirt-devel] [ OST Failure Report ] [ oVirt Master ] [ 14-12-2017 ] [ 004_basic_sanity.disk_operations ] Date: Thu, 14 Dec 2017 13:22:31 +0100 Message-ID: In-Reply-To: E8C90687-357B-46C4-902D-2FB5077F49B9@redhat.com --===============6930952422421919488== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable This is a multi-part message in MIME format. --------------7C4A69F5EC47C0DE9F67DB09 Content-Type: text/plain; charset=3Dutf-8 Content-Transfer-Encoding: 8bit On 12/14/2017 01:12 PM, Michal Skrivanek wrote: > >> On 14 Dec 2017, at 13:00, Dafna Ron > > wrote: >> >> Hi, >> >> We have a failure on basic suite on test: >> 004_basic_sanity.disk_operations >> >> I think that we query a snapshot that was already deleted >> successfully and report the snapshot as gone. and that is because of >> a different error in update vm query which happens before. >> >> * >> Link and headline of suspected patches: >> https://gerrit.ovirt.org/#/c/85168/ - core: Prevent retry lease >> hotplag in case of failure. >> >> Link to Job: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393 >> >> Link to all logs: >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4393/artif= act/ >> >> (Relevant) error snippet from the log: >> >> * >> >> vdsm:=C2=A0** >> ** >> 2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) >> [jsonrpc.JsonRpcServer] Internal server error (__init__:611) >> Traceback (most recent call last): >> =C2=A0 File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", li= ne >> 606, in _handle_request >> =C2=A0=C2=A0=C2=A0 res =3D method(**params) >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line >> 201, in _dynamicMethod >> =C2=A0=C2=A0=C2=A0 result =3D fn(*methodArgs) >> =C2=A0 File "", line 2, in getAllVmStats >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line >> 48, in method >> =C2=A0=C2=A0=C2=A0 ret =3D func(*args, **kwargs) >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1342, in >> getAllVmStats >> =C2=A0=C2=A0=C2=A0 statsList =3D self._cif.getAllVmStats() >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 51= 8, >> in getAllVmStats >> =C2=A0=C2=A0=C2=A0 return [v.getStats() for v in self.vmContainer.values= ()] >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 169= 9, >> in getStats >> =C2=A0=C2=A0=C2=A0 oga_stats =3D self._getGuestStats() >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 189= 5, >> in _getGuestStats >> =C2=A0=C2=A0=C2=A0 self._update_guest_disk_mapping() >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 190= 9, >> in _update_guest_disk_mapping >> =C2=A0=C2=A0=C2=A0 self._sync_metadata() >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 499= 5, >> in _sync_metadata >> =C2=A0=C2=A0=C2=A0 self._md_desc.dump(self._dom) >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", li= ne >> 477, in dump >> =C2=A0=C2=A0=C2=A0 dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT, >> =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", >> line 47, in __getattr__ >> =C2=A0=C2=A0=C2=A0 % self.vmid) >> NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not >> defined yet or was undefined >> ** > > it doesn=E2=80=99t seem to be relevant to this failure, but it deserves a= fix > nevertheless > Francesco? > Smells like a race on shutdown. -- = Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh --------------7C4A69F5EC47C0DE9F67DB09 Content-Type: text/html; charset=3Dutf-8 Content-Transfer-Encoding: 8bit
On 12/14/2017 01:12 PM, Michal Skrivanek wrote:

On 14 Dec 2017, at 13:00, Dafna Ron <dron(a)redhat.com> wrote:

Hi,

We have a failure on basic suite on test: 004_basic_sanity.disk_operations

I think that we query a snapshot that was already deleted successfully and report the snapshot as gone. and that is because of a different error in update vm query which happens before.

Link and headline of suspected patches:
https://gerrit.ovirt.org/#/c/851= 68/ - core: Prevent retry lease hotplag in case of failure.

Link to Job:
http://jenkins.ovirt.org/job/ovi= rt-master_change-queue-tester/4393

Link to all logs:
http://jenkins.ovirt.org/job/o= virt-master_change-queue-tester/4393/artifact/

(Relevant) error snippet from the log:
<error>


vdsm:=C2=A0

2017-12-14 02:34:45,222-0500 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611)
Traceback (most recent call last):
=C2=A0 File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request
=C2=A0=C2=A0=C2=A0 res =3D method(**params)
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod
=C2=A0=C2=A0=C2=A0 result =3D fn(*methodArgs)
=C2=A0 File "<string>", line 2, in getAllVmStats
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
=C2=A0=C2=A0=C2=A0 ret =3D func(*args, **kwargs)
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/API.py= ", line 1342, in getAllVmStats
=C2=A0=C2=A0=C2=A0 statsList =3D self._cif.getAllVmStats(= )
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 518, in getAllVmStats
=C2=A0=C2=A0=C2=A0 return [v.getStats() for v in self.vmContainer.values()]
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1699, in getStats
=C2=A0=C2=A0=C2=A0 oga_stats =3D self._getGuestStats() =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1895, in _getGuestStats
=C2=A0=C2=A0=C2=A0 self._update_guest_disk_mapping()
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1909, in _update_guest_disk_mapping
=C2=A0=C2=A0=C2=A0 self._sync_metadata()
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4995, in _sync_metadata
=C2=A0=C2=A0=C2=A0 self._md_desc.dump(self._dom)
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line 477, in dump
=C2=A0=C2=A0=C2=A0 dom.setMetadata(libvirt.VIR_DOMAIN_METADATA_ELEMENT,
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__
=C2=A0=C2=A0=C2=A0 % self.vmid)
NotConnectedError: VM '7cab7e5a-cb12-4977-ac4f-65218532df7e' was not defined yet or was undefined

it doesn=E2=80=99t seem to be relevant to this failure, but it dese= rves a fix nevertheless
Francesco?


Smells like a race on shutdown.

-- =

Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
--------------7C4A69F5EC47C0DE9F67DB09-- --===============6930952422421919488== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" VGhpcyBpcyBhIG11bHRpLXBhcnQgbWVzc2FnZSBpbiBNSU1FIGZvcm1hdC4KLS0tLS0tLS0tLS0t LS03QzRBNjlGNUVDNDdDMERFOUY2N0RCMDkKQ29udGVudC1UeXBlOiB0ZXh0L3BsYWluOyBjaGFy c2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IDhiaXQKCgpPbiAxMi8xNC8yMDE3 IDAxOjEyIFBNLCBNaWNoYWwgU2tyaXZhbmVrIHdyb3RlOgo+Cj4+IE9uIDE0IERlYyAyMDE3LCBh dCAxMzowMCwgRGFmbmEgUm9uIDxkcm9uQHJlZGhhdC5jb20KPj4gPG1haWx0bzpkcm9uQHJlZGhh dC5jb20+PiB3cm90ZToKPj4KPj4gSGksCj4+Cj4+IFdlIGhhdmUgYSBmYWlsdXJlIG9uIGJhc2lj IHN1aXRlIG9uIHRlc3Q6Cj4+IDAwNF9iYXNpY19zYW5pdHkuZGlza19vcGVyYXRpb25zCj4+Cj4+ IEkgdGhpbmsgdGhhdCB3ZSBxdWVyeSBhIHNuYXBzaG90IHRoYXQgd2FzIGFscmVhZHkgZGVsZXRl ZAo+PiBzdWNjZXNzZnVsbHkgYW5kIHJlcG9ydCB0aGUgc25hcHNob3QgYXMgZ29uZS4gYW5kIHRo YXQgaXMgYmVjYXVzZSBvZgo+PiBhIGRpZmZlcmVudCBlcnJvciBpbiB1cGRhdGUgdm0gcXVlcnkg d2hpY2ggaGFwcGVucyBiZWZvcmUuCj4+Cj4+ICoKPj4gTGluayBhbmQgaGVhZGxpbmUgb2Ygc3Vz cGVjdGVkIHBhdGNoZXM6Cj4+IGh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2MvODUxNjgvIC0g Y29yZTogUHJldmVudCByZXRyeSBsZWFzZQo+PiBob3RwbGFnIGluIGNhc2Ugb2YgZmFpbHVyZS4K Pj4KPj4gTGluayB0byBKb2I6Cj4+IGh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQt bWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDM5Mwo+Pgo+PiBMaW5rIHRvIGFsbCBsb2dzOgo+ PiBodHRwOi8vamVua2lucy5vdmlydC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUt dGVzdGVyLzQzOTMvYXJ0aWZhY3QvCj4+Cj4+IChSZWxldmFudCkgZXJyb3Igc25pcHBldCBmcm9t IHRoZSBsb2c6Cj4+IDxlcnJvcj4KPj4gKgo+Pgo+PiB2ZHNtOsKgKioKPj4gKioKPj4gMjAxNy0x Mi0xNCAwMjozNDo0NSwyMjItMDUwMCBFUlJPUiAoanNvbnJwYy83KQo+PiBbanNvbnJwYy5Kc29u UnBjU2VydmVyXSBJbnRlcm5hbCBzZXJ2ZXIgZXJyb3IgKF9faW5pdF9fOjYxMSkKPj4gVHJhY2Vi YWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgo+PiDCoCBGaWxlICIvdXNyL2xpYi9weXRob24y Ljcvc2l0ZS1wYWNrYWdlcy95YWpzb25ycGMvX19pbml0X18ucHkiLCBsaW5lCj4+IDYwNiwgaW4g X2hhbmRsZV9yZXF1ZXN0Cj4+IMKgwqDCoCByZXMgPSBtZXRob2QoKipwYXJhbXMpCj4+IMKgIEZp bGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vcnBjL0JyaWRnZS5weSIs IGxpbmUKPj4gMjAxLCBpbiBfZHluYW1pY01ldGhvZAo+PiDCoMKgwqAgcmVzdWx0ID0gZm4oKm1l dGhvZEFyZ3MpCj4+IMKgIEZpbGUgIjxzdHJpbmc+IiwgbGluZSAyLCBpbiBnZXRBbGxWbVN0YXRz Cj4+IMKgIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vY29tbW9u L2FwaS5weSIsIGxpbmUKPj4gNDgsIGluIG1ldGhvZAo+PiDCoMKgwqAgcmV0ID0gZnVuYygqYXJn cywgKiprd2FyZ3MpCj4+IMKgIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2Vz L3Zkc20vQVBJLnB5IiwgbGluZSAxMzQyLCBpbgo+PiBnZXRBbGxWbVN0YXRzCj4+IMKgwqDCoCBz dGF0c0xpc3QgPSBzZWxmLl9jaWYuZ2V0QWxsVm1TdGF0cygpCj4+IMKgIEZpbGUgIi91c3IvbGli L3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vY2xpZW50SUYucHkiLCBsaW5lIDUxOCwKPj4g aW4gZ2V0QWxsVm1TdGF0cwo+PiDCoMKgwqAgcmV0dXJuIFt2LmdldFN0YXRzKCkgZm9yIHYgaW4g c2VsZi52bUNvbnRhaW5lci52YWx1ZXMoKV0KPj4gwqAgRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43 L3NpdGUtcGFja2FnZXMvdmRzbS92aXJ0L3ZtLnB5IiwgbGluZSAxNjk5LAo+PiBpbiBnZXRTdGF0 cwo+PiDCoMKgwqAgb2dhX3N0YXRzID0gc2VsZi5fZ2V0R3Vlc3RTdGF0cygpCj4+IMKgIEZpbGUg Ii91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vdmlydC92bS5weSIsIGxpbmUg MTg5NSwKPj4gaW4gX2dldEd1ZXN0U3RhdHMKPj4gwqDCoMKgIHNlbGYuX3VwZGF0ZV9ndWVzdF9k aXNrX21hcHBpbmcoKQo+PiDCoCBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdl cy92ZHNtL3ZpcnQvdm0ucHkiLCBsaW5lIDE5MDksCj4+IGluIF91cGRhdGVfZ3Vlc3RfZGlza19t YXBwaW5nCj4+IMKgwqDCoCBzZWxmLl9zeW5jX21ldGFkYXRhKCkKPj4gwqAgRmlsZSAiL3Vzci9s aWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvdmRzbS92aXJ0L3ZtLnB5IiwgbGluZSA0OTk1LAo+ PiBpbiBfc3luY19tZXRhZGF0YQo+PiDCoMKgwqAgc2VsZi5fbWRfZGVzYy5kdW1wKHNlbGYuX2Rv bSkKPj4gwqAgRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvdmRzbS92aXJ0 L21ldGFkYXRhLnB5IiwgbGluZQo+PiA0NzcsIGluIGR1bXAKPj4gwqDCoMKgIGRvbS5zZXRNZXRh ZGF0YShsaWJ2aXJ0LlZJUl9ET01BSU5fTUVUQURBVEFfRUxFTUVOVCwKPj4gwqAgRmlsZSAiL3Vz ci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvdmRzbS92aXJ0L3ZpcmRvbWFpbi5weSIsCj4+ IGxpbmUgNDcsIGluIF9fZ2V0YXR0cl9fCj4+IMKgwqDCoCAlIHNlbGYudm1pZCkKPj4gTm90Q29u bmVjdGVkRXJyb3I6IFZNICc3Y2FiN2U1YS1jYjEyLTQ5NzctYWM0Zi02NTIxODUzMmRmN2UnIHdh cyBub3QKPj4gZGVmaW5lZCB5ZXQgb3Igd2FzIHVuZGVmaW5lZAo+PiAqKgo+Cj4gaXQgZG9lc27i gJl0IHNlZW0gdG8gYmUgcmVsZXZhbnQgdG8gdGhpcyBmYWlsdXJlLCBidXQgaXQgZGVzZXJ2ZXMg YSBmaXgKPiBuZXZlcnRoZWxlc3MKPiBGcmFuY2VzY28/Cj4KClNtZWxscyBsaWtlIGEgcmFjZSBv biBzaHV0ZG93bi4KCi0tIApGcmFuY2VzY28gUm9tYW5pClNlbmlvciBTVyBFbmcuLCBWaXJ0dWFs aXphdGlvbiBSJkQKUmVkIEhhdApJUkM6IGZyb21hbmkgZ2l0aHViOiBAZnJvbWFuaXJoCgoKLS0t LS0tLS0tLS0tLS03QzRBNjlGNUVDNDdDMERFOUY2N0RCMDkKQ29udGVudC1UeXBlOiB0ZXh0L2h0 bWw7IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogOGJpdAoKPGh0bWw+ CiAgPGhlYWQ+CiAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl eHQvaHRtbDsgY2hhcnNldD11dGYtOCI+CiAgPC9oZWFkPgogIDxib2R5IHRleHQ9IiMwMDAwMDAi IGJnY29sb3I9IiNGRkZGRkYiPgogICAgPGJyPgogICAgPGRpdiBjbGFzcz0ibW96LWNpdGUtcHJl Zml4Ij5PbiAxMi8xNC8yMDE3IDAxOjEyIFBNLCBNaWNoYWwKICAgICAgU2tyaXZhbmVrIHdyb3Rl Ojxicj4KICAgIDwvZGl2PgogICAgPGJsb2NrcXVvdGUgdHlwZT0iY2l0ZSIKICAgICAgY2l0ZT0i bWlkOkU4QzkwNjg3LTM1N0ItNDZDNC05MDJELTJGQjUwNzdGNDlCOUByZWRoYXQuY29tIj4KICAg ICAgPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNo YXJzZXQ9dXRmLTgiPgogICAgICA8YnIgY2xhc3M9IiI+CiAgICAgIDxkaXY+CiAgICAgICAgPGJs b2NrcXVvdGUgdHlwZT0iY2l0ZSIgY2xhc3M9IiI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSIiPk9u IDE0IERlYyAyMDE3LCBhdCAxMzowMCwgRGFmbmEgUm9uICZsdDs8YQogICAgICAgICAgICAgIGhy ZWY9Im1haWx0bzpkcm9uQHJlZGhhdC5jb20iIGNsYXNzPSIiCiAgICAgICAgICAgICAgbW96LWRv LW5vdC1zZW5kPSJ0cnVlIj5kcm9uQHJlZGhhdC5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj4KICAg ICAgICAgIDxiciBjbGFzcz0iQXBwbGUtaW50ZXJjaGFuZ2UtbmV3bGluZSI+CiAgICAgICAgICA8 ZGl2IGNsYXNzPSIiPgogICAgICAgICAgICA8bWV0YSBodHRwLWVxdWl2PSJjb250ZW50LXR5cGUi IGNvbnRlbnQ9InRleHQvaHRtbDsKICAgICAgICAgICAgICBjaGFyc2V0PXV0Zi04IiBjbGFzcz0i Ij4KICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAwIiBiZ2NvbG9yPSIjRkZGRkZGIiBjbGFz cz0iIj4KICAgICAgICAgICAgICA8cCBjbGFzcz0iIj5IaSwgPGJyIGNsYXNzPSIiPgogICAgICAg ICAgICAgIDwvcD4KICAgICAgICAgICAgICA8cCBjbGFzcz0iIj5XZSBoYXZlIGEgZmFpbHVyZSBv biBiYXNpYyBzdWl0ZSBvbiB0ZXN0OgogICAgICAgICAgICAgICAgMDA0X2Jhc2ljX3Nhbml0eS5k aXNrX29wZXJhdGlvbnM8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgPC9wPgogICAgICAgICAg ICAgIEkgdGhpbmsgdGhhdCB3ZSBxdWVyeSBhIHNuYXBzaG90IHRoYXQgd2FzIGFscmVhZHkgZGVs ZXRlZAogICAgICAgICAgICAgIHN1Y2Nlc3NmdWxseSBhbmQgcmVwb3J0IHRoZSBzbmFwc2hvdCBh cyBnb25lLiBhbmQgdGhhdCBpcwogICAgICAgICAgICAgIGJlY2F1c2Ugb2YgYSBkaWZmZXJlbnQg ZXJyb3IgaW4gdXBkYXRlIHZtIHF1ZXJ5IHdoaWNoCiAgICAgICAgICAgICAgaGFwcGVucyBiZWZv cmUuIDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAg ICAgPGIgc3R5bGU9ImZvbnQtd2VpZ2h0Om5vcm1hbDsiCiAgICAgICAgICAgICAgICBpZD0iZG9j cy1pbnRlcm5hbC1ndWlkLTU4NTliN2ExLTU0ZGQtODJmZS04ZmM4LThmYWI1ZTk4MTg2OCIKICAg ICAgICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgPGRpdiBzdHlsZT0ibGluZS1o ZWlnaHQ6IDEuMzg7IG1hcmdpbi10b3A6IDBwdDsKICAgICAgICAgICAgICAgICAgbWFyZ2luLWJv dHRvbTogMHB0OyIgY2xhc3M9IiI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFwdDsgZm9udC1m YW1pbHk6IEFyaWFsOyBiYWNrZ3JvdW5kLWNvbG9yOiB0cmFuc3BhcmVudDsgZm9udC13ZWlnaHQ6 IDQwMDsgZm9udC1zdHlsZTogbm9ybWFsOyBmb250LXZhcmlhbnQtbGlnYXR1cmVzOiBub3JtYWw7 IGZvbnQtdmFyaWFudC1jYXBzOiBub3JtYWw7IGZvbnQtdmFyaWFudC1lYXN0LWFzaWFuOiBub3Jt YWw7IGZvbnQtdmFyaWFudC1wb3NpdGlvbjogbm9ybWFsOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7 IHZlcnRpY2FsLWFsaWduOiBiYXNlbGluZTsgd2hpdGUtc3BhY2U6IHByZS13cmFwOyIgY2xhc3M9 IiI+TGluayBhbmQgaGVhZGxpbmUgb2Ygc3VzcGVjdGVkIHBhdGNoZXM6IDwvc3Bhbj48L2Rpdj4K ICAgICAgICAgICAgICAgIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiCiAgICAgICAg ICAgICAgICAgIGhyZWY9Imh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2MvODUxNjgvIgogICAg ICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiPmh0dHBzOi8vZ2Vycml0Lm92aXJ0 Lm9yZy8jL2MvODUxNjgvPC9hPgogICAgICAgICAgICAgICAgLSBjb3JlOiBQcmV2ZW50IHJldHJ5 IGxlYXNlIGhvdHBsYWcgaW4gY2FzZSBvZiBmYWlsdXJlLjxicgogICAgICAgICAgICAgICAgICBj bGFzcz0iIj4KICAgICAgICAgICAgICAgIDxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgIDxk aXYgc3R5bGU9ImxpbmUtaGVpZ2h0OiAxLjM4OyBtYXJnaW4tdG9wOiAwcHQ7CiAgICAgICAgICAg ICAgICAgIG1hcmdpbi1ib3R0b206IDBwdDsiIGNsYXNzPSIiPjxzcGFuIHN0eWxlPSJmb250LXNp emU6IDExcHQ7IGZvbnQtZmFtaWx5OiBBcmlhbDsgYmFja2dyb3VuZC1jb2xvcjogdHJhbnNwYXJl bnQ7IGZvbnQtd2VpZ2h0OiA0MDA7IGZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWxp Z2F0dXJlczogbm9ybWFsOyBmb250LXZhcmlhbnQtY2Fwczogbm9ybWFsOyBmb250LXZhcmlhbnQt ZWFzdC1hc2lhbjogbm9ybWFsOyBmb250LXZhcmlhbnQtcG9zaXRpb246IG5vcm1hbDsgdGV4dC1k ZWNvcmF0aW9uOiBub25lOyB2ZXJ0aWNhbC1hbGlnbjogYmFzZWxpbmU7IHdoaXRlLXNwYWNlOiBw cmUtd3JhcDsiIGNsYXNzPSIiPkxpbmsgdG8gSm9iOjwvc3Bhbj48L2Rpdj4KICAgICAgICAgICAg ICAgIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiCmhyZWY9Imh0dHA6Ly9qZW5raW5z Lm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFzdGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDM5MyIKICAg ICAgICAgICAgICAgICAgbW96LWRvLW5vdC1zZW5kPSJ0cnVlIj5odHRwOi8vamVua2lucy5vdmly dC5vcmcvam9iL292aXJ0LW1hc3Rlcl9jaGFuZ2UtcXVldWUtdGVzdGVyLzQzOTM8L2E+PGJyCiAg ICAgICAgICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgPGJyIGNsYXNzPSIiPgog ICAgICAgICAgICAgICAgPGRpdiBzdHlsZT0ibGluZS1oZWlnaHQ6IDEuMzg7IG1hcmdpbi10b3A6 IDBwdDsKICAgICAgICAgICAgICAgICAgbWFyZ2luLWJvdHRvbTogMHB0OyIgY2xhc3M9IiI+PHNw YW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFwdDsgZm9udC1mYW1pbHk6IEFyaWFsOyBiYWNrZ3JvdW5k LWNvbG9yOiB0cmFuc3BhcmVudDsgZm9udC13ZWlnaHQ6IDQwMDsgZm9udC1zdHlsZTogbm9ybWFs OyBmb250LXZhcmlhbnQtbGlnYXR1cmVzOiBub3JtYWw7IGZvbnQtdmFyaWFudC1jYXBzOiBub3Jt YWw7IGZvbnQtdmFyaWFudC1lYXN0LWFzaWFuOiBub3JtYWw7IGZvbnQtdmFyaWFudC1wb3NpdGlv bjogbm9ybWFsOyB0ZXh0LWRlY29yYXRpb246IG5vbmU7IHZlcnRpY2FsLWFsaWduOiBiYXNlbGlu ZTsgd2hpdGUtc3BhY2U6IHByZS13cmFwOyIgY2xhc3M9IiI+TGluayB0byBhbGwgbG9nczo8L3Nw YW4+PC9kaXY+CiAgICAgICAgICAgICAgICA8dHQ+PGEgY2xhc3M9Im1vei10eHQtbGluay1mcmVl dGV4dCIKaHJlZj0iaHR0cDovL2plbmtpbnMub3ZpcnQub3JnL2pvYi9vdmlydC1tYXN0ZXJfY2hh bmdlLXF1ZXVlLXRlc3Rlci80MzkzL2FydGlmYWN0LyIKICAgICAgICAgICAgICAgICAgICBtb3ot ZG8tbm90LXNlbmQ9InRydWUiPmh0dHA6Ly9qZW5raW5zLm92aXJ0Lm9yZy9qb2Ivb3ZpcnQtbWFz dGVyX2NoYW5nZS1xdWV1ZS10ZXN0ZXIvNDM5My9hcnRpZmFjdC88L2E+PC90dD48YnIKICAgICAg ICAgICAgICAgICAgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAg ICAgICAgICAgICA8ZGl2IHN0eWxlPSJsaW5lLWhlaWdodDogMS4zODsgbWFyZ2luLXRvcDogMHB0 OwogICAgICAgICAgICAgICAgICBtYXJnaW4tYm90dG9tOiAwcHQ7IiBjbGFzcz0iIj48c3BhbiBz dHlsZT0iZm9udC1zaXplOiAxMXB0OyBmb250LWZhbWlseTogQXJpYWw7IGJhY2tncm91bmQtY29s b3I6IHRyYW5zcGFyZW50OyBmb250LXdlaWdodDogNDAwOyBmb250LXN0eWxlOiBub3JtYWw7IGZv bnQtdmFyaWFudC1saWdhdHVyZXM6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsg Zm9udC12YXJpYW50LWVhc3QtYXNpYW46IG5vcm1hbDsgZm9udC12YXJpYW50LXBvc2l0aW9uOiBu b3JtYWw7IHRleHQtZGVjb3JhdGlvbjogbm9uZTsgdmVydGljYWwtYWxpZ246IGJhc2VsaW5lOyB3 aGl0ZS1zcGFjZTogcHJlLXdyYXA7IiBjbGFzcz0iIj4oUmVsZXZhbnQpIGVycm9yIHNuaXBwZXQg ZnJvbSB0aGUgbG9nOiA8L3NwYW4+PC9kaXY+CiAgICAgICAgICAgICAgICA8ZGl2IHN0eWxlPSJs aW5lLWhlaWdodDogMS4zODsgbWFyZ2luLXRvcDogMHB0OwogICAgICAgICAgICAgICAgICBtYXJn aW4tYm90dG9tOiAwcHQ7IiBjbGFzcz0iIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMXB0OyBm b250LWZhbWlseTogQXJpYWw7IGJhY2tncm91bmQtY29sb3I6IHRyYW5zcGFyZW50OyBmb250LXdl aWdodDogNDAwOyBmb250LXN0eWxlOiBub3JtYWw7IGZvbnQtdmFyaWFudC1saWdhdHVyZXM6IG5v cm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgZm9udC12YXJpYW50LWVhc3QtYXNpYW46 IG5vcm1hbDsgZm9udC12YXJpYW50LXBvc2l0aW9uOiBub3JtYWw7IHRleHQtZGVjb3JhdGlvbjog bm9uZTsgdmVydGljYWwtYWxpZ246IGJhc2VsaW5lOyB3aGl0ZS1zcGFjZTogcHJlLXdyYXA7IiBj bGFzcz0iIj4mbHQ7ZXJyb3ImZ3Q7PC9zcGFuPjwvZGl2PgogICAgICAgICAgICAgICAgPGIgY2xh c3M9IiI+PGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICA8YnIgY2xhc3M9IiI+CiAgICAg ICAgICAgICAgICAgIHZkc206wqA8L2I+PC9iPjwvZGl2PgogICAgICAgICAgPC9kaXY+CiAgICAg ICAgPC9ibG9ja3F1b3RlPgogICAgICAgIDxibG9ja3F1b3RlIHR5cGU9ImNpdGUiIGNsYXNzPSIi PgogICAgICAgICAgPGRpdiBjbGFzcz0iIj4KICAgICAgICAgICAgPGRpdiB0ZXh0PSIjMDAwMDAw IiBiZ2NvbG9yPSIjRkZGRkZGIiBjbGFzcz0iIj48YgogICAgICAgICAgICAgICAgc3R5bGU9ImZv bnQtd2VpZ2h0Om5vcm1hbDsiCiAgICAgICAgICAgICAgICBpZD0iZG9jcy1pbnRlcm5hbC1ndWlk LTU4NTliN2ExLTU0ZGQtODJmZS04ZmM4LThmYWI1ZTk4MTg2OCIKICAgICAgICAgICAgICAgIGNs YXNzPSIiPjxiIGNsYXNzPSIiPiA8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIDIwMTct MTItMTQgMDI6MzQ6NDUsMjIyLTA1MDAgRVJST1IgKGpzb25ycGMvNykKICAgICAgICAgICAgICAg ICAgW2pzb25ycGMuSnNvblJwY1NlcnZlcl0gSW50ZXJuYWwgc2VydmVyIGVycm9yCiAgICAgICAg ICAgICAgICAgIChfX2luaXRfXzo2MTEpPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICBU cmFjZWJhY2sgKG1vc3QgcmVjZW50IGNhbGwgbGFzdCk6PGJyIGNsYXNzPSIiPgogICAgICAgICAg ICAgICAgICDCoCBGaWxlCiAgICAgICAgICAgICAgICAgICIvdXNyL2xpYi9weXRob24yLjcvc2l0 ZS1wYWNrYWdlcy95YWpzb25ycGMvX19pbml0X18ucHkiLAogICAgICAgICAgICAgICAgICBsaW5l IDYwNiwgaW4gX2hhbmRsZV9yZXF1ZXN0PGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICDC oMKgwqAgcmVzID0gbWV0aG9kKCoqcGFyYW1zKTxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAg ICAgwqAgRmlsZQogICAgICAgICAgICAgICAgICAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFj a2FnZXMvdmRzbS9ycGMvQnJpZGdlLnB5IiwKICAgICAgICAgICAgICAgICAgbGluZSAyMDEsIGlu IF9keW5hbWljTWV0aG9kPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICDCoMKgwqAgcmVz dWx0ID0gZm4oKm1ldGhvZEFyZ3MpPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICDCoCBG aWxlICImbHQ7c3RyaW5nJmd0OyIsIGxpbmUgMiwgaW4gZ2V0QWxsVm1TdGF0czxicgogICAgICAg ICAgICAgICAgICAgIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICDCoCBGaWxlCiAgICAgICAg ICAgICAgICAgICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNtL2NvbW1vbi9h cGkucHkiLAogICAgICAgICAgICAgICAgICBsaW5lIDQ4LCBpbiBtZXRob2Q8YnIgY2xhc3M9IiI+ CiAgICAgICAgICAgICAgICAgIMKgwqDCoCByZXQgPSBmdW5jKCphcmdzLCAqKmt3YXJncyk8YnIg Y2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9z aXRlLXBhY2thZ2VzL3Zkc20vQVBJLnB5IiwKICAgICAgICAgICAgICAgICAgbGluZSAxMzQyLCBp biBnZXRBbGxWbVN0YXRzPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICDCoMKgwqAgc3Rh dHNMaXN0ID0gc2VsZi5fY2lmLmdldEFsbFZtU3RhdHMoKTxiciBjbGFzcz0iIj4KICAgICAgICAg ICAgICAgICAgwqAgRmlsZQogICAgICAgICAgICAgICAgICAiL3Vzci9saWIvcHl0aG9uMi43L3Np dGUtcGFja2FnZXMvdmRzbS9jbGllbnRJRi5weSIsCiAgICAgICAgICAgICAgICAgIGxpbmUgNTE4 LCBpbiBnZXRBbGxWbVN0YXRzPGJyIGNsYXNzPSIiPgogICAgICAgICAgICAgICAgICDCoMKgwqAg cmV0dXJuIFt2LmdldFN0YXRzKCkgZm9yIHYgaW4KICAgICAgICAgICAgICAgICAgc2VsZi52bUNv bnRhaW5lci52YWx1ZXMoKV08YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgIEZpbGUK ICAgICAgICAgICAgICAgICAgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20v dmlydC92bS5weSIsCiAgICAgICAgICAgICAgICAgIGxpbmUgMTY5OSwgaW4gZ2V0U3RhdHM8YnIg Y2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgwqDCoCBvZ2Ffc3RhdHMgPSBzZWxmLl9nZXRH dWVzdFN0YXRzKCk8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgIEZpbGUKICAgICAg ICAgICAgICAgICAgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vdmlydC92 bS5weSIsCiAgICAgICAgICAgICAgICAgIGxpbmUgMTg5NSwgaW4gX2dldEd1ZXN0U3RhdHM8YnIg Y2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgwqDCoCBzZWxmLl91cGRhdGVfZ3Vlc3RfZGlz a19tYXBwaW5nKCk8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgIEZpbGUKICAgICAg ICAgICAgICAgICAgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vdmlydC92 bS5weSIsCiAgICAgICAgICAgICAgICAgIGxpbmUgMTkwOSwgaW4gX3VwZGF0ZV9ndWVzdF9kaXNr X21hcHBpbmc8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgwqDCoCBzZWxmLl9zeW5j X21ldGFkYXRhKCk8YnIgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgIEZpbGUKICAgICAg ICAgICAgICAgICAgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vdmlydC92 bS5weSIsCiAgICAgICAgICAgICAgICAgIGxpbmUgNDk5NSwgaW4gX3N5bmNfbWV0YWRhdGE8YnIg Y2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgwqDCoCBzZWxmLl9tZF9kZXNjLmR1bXAoc2Vs Zi5fZG9tKTxiciBjbGFzcz0iIj4KICAgICAgICAgICAgICAgICAgwqAgRmlsZQogICAgICAgICAg ICAgICAgICAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUtcGFja2FnZXMvdmRzbS92aXJ0L21ldGFk YXRhLnB5IiwKICAgICAgICAgICAgICAgICAgbGluZSA0NzcsIGluIGR1bXA8YnIgY2xhc3M9IiI+ CiAgICAgICAgICAgICAgICAgIMKgwqDCoAogICAgICAgICAgICAgICAgICBkb20uc2V0TWV0YWRh dGEobGlidmlydC5WSVJfRE9NQUlOX01FVEFEQVRBX0VMRU1FTlQsPGJyCiAgICAgICAgICAgICAg ICAgICAgY2xhc3M9IiI+CiAgICAgICAgICAgICAgICAgIMKgIEZpbGUKICAgICAgICAgICAgICAg ICAgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vdmlydC92aXJkb21haW4u cHkiLAogICAgICAgICAgICAgICAgICBsaW5lIDQ3LCBpbiBfX2dldGF0dHJfXzxiciBjbGFzcz0i Ij4KICAgICAgICAgICAgICAgICAgwqDCoMKgICUgc2VsZi52bWlkKTxiciBjbGFzcz0iIj4KICAg ICAgICAgICAgICAgICAgTm90Q29ubmVjdGVkRXJyb3I6IFZNCiAgICAgICAgICAgICAgICAgICc3 Y2FiN2U1YS1jYjEyLTQ5NzctYWM0Zi02NTIxODUzMmRmN2UnIHdhcyBub3QgZGVmaW5lZAogICAg ICAgICAgICAgICAgICB5ZXQgb3Igd2FzIHVuZGVmaW5lZDxiciBjbGFzcz0iIj4KICAgICAgICAg ICAgICAgIDwvYj48L2I+PC9kaXY+CiAgICAgICAgICA8L2Rpdj4KICAgICAgICA8L2Jsb2NrcXVv dGU+CiAgICAgICAgPGRpdj48YnIgY2xhc3M9IiI+CiAgICAgICAgPC9kaXY+CiAgICAgICAgaXQg ZG9lc27igJl0IHNlZW0gdG8gYmUgcmVsZXZhbnQgdG8gdGhpcyBmYWlsdXJlLCBidXQgaXQgZGVz ZXJ2ZXMKICAgICAgICBhIGZpeCBuZXZlcnRoZWxlc3M8L2Rpdj4KICAgICAgPGRpdj5GcmFuY2Vz Y28/PC9kaXY+CiAgICAgIDxkaXY+PGJyIGNsYXNzPSIiPgogICAgICA8L2Rpdj4KICAgIDwvYmxv Y2txdW90ZT4KICAgIDxicj4KICAgIFNtZWxscyBsaWtlIGEgcmFjZSBvbiBzaHV0ZG93bi48YnI+ CiAgICA8YnI+CiAgICA8cHJlIGNsYXNzPSJtb3otc2lnbmF0dXJlIiBjb2xzPSI3MiI+LS0gCkZy YW5jZXNjbyBSb21hbmkKU2VuaW9yIFNXIEVuZy4sIFZpcnR1YWxpemF0aW9uIFImYW1wO0QKUmVk IEhhdApJUkM6IGZyb21hbmkgZ2l0aHViOiBAZnJvbWFuaXJoPC9wcmU+CiAgPC9ib2R5Pgo8L2h0 bWw+CgotLS0tLS0tLS0tLS0tLTdDNEE2OUY1RUM0N0MwREU5RjY3REIwOS0tCg== --===============6930952422421919488==--