From chapelle at cines.fr Tue Jun 7 10:59:07 2016
Content-Type: multipart/mixed; boundary="===============6350193133577545074=="
MIME-Version: 1.0
From: =?utf-8?q?J=C3=A9r=C3=B4me_Chapelle_=3Cchapelle_at_cines=2Efr=3E?=
To: users at ovirt.org
Subject: [ovirt-users] PCI Passthrough problem
Date: Tue, 07 Jun 2016 16:50:35 +0200
Message-ID: <244701585.366646.1465311035702.JavaMail.zimbra@cines.fr>
--===============6350193133577545074==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
------=3D_Part_366645_1002603632.1465311035701
Content-Type: text/plain; charset=3Dutf-8
Content-Transfer-Encoding: quoted-printable
We are no more able to give a pci device to one VM.=3D20
Here is the process we used to assign one device to one VM:=3D20
- Click on the VM, and then on the tab "Host devices"=3D20
- In the tab, click on "pin to host" and select the host that will give the=
=3D
pci device=3D20
- Click on "add device", in the window appear all the pci devices: choose t=
=3D
he pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then click=
=3D
on the gray arrow to assign this device to the VM=3D20
- Click on ok=3D20
- boot the VM.=3D20
The problem is that in the window that appears, there is not a single pci d=
=3D
evice listed (there should be many listed).=3D20
I checked first: the IOMMU setting is set in the kernel.=3D20
Our environment:=3D20
We have two HV running CentOs7.=3D20
Each of them have three fiber channel cards. First card is connected to a d=
=3D
isk array, second one to the first tape library and third one to another ta=
=3D
pe library.=3D20
We used to give the second card to a VM1, and the third card to another VM2=
=3D
. Both VMs ran on one HV or on the other in the past: everything was fine.=
=3D
=3D20
What happened:=3D20
Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. We u=
=3D
pgraded then restarted the HV1.=3D20
Then I edited the second VM2 in order to remove card from HV1. Then I tried=
=3D
to add the card from HV2 to this VM2: the list of "host devices" is empty.=
=3D
=3D20
I started the VM1: it works fine (I didn't change anything on it).=3D20
--=3D20
=3D09
=3D09
=3D09
=3D09=3D09J=3DC3=3DA9r=3DC3=3DB4me Chapelle=3D20
=3D09=3D09C entre I nformatique N ational de l' E nseignement S up=3DC3=3DA=
9rieur=3D
=3D20
950 rue de Saint Priest=3D20
34 097 M ONTPELLIER Cedex 5=3D20
=3D09=3D09T=3DC3=3DA9l. =3D0904 67 14 14 14=3D20
=3D09=3D09eMail =3D09chapelle(a)cines.fr=3D20
------=3D_Part_366645_1002603632.1465311035701
Content-Type: text/html; charset=3Dutf-8
Content-Transfer-Encoding: quoted-printable
We are no more able to give a pci device to one VM.
Here is the process we used to assign one =
de=3D
vice to one VM:
- Click on the VM, and=
t=3D
hen on the tab "Host devices"
- In the tab, click on "pin to =
=3D
host" and select the host that will give the pci device
- Click on "add device", in the window appear all the p=
ci=3D
devices: choose the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00=
=3D
_1) and then click on the gray arrow to assign this device to the VM
- Click on ok
- boot the VM.
The problem is that =
in=3D
the window that appears, there is not a single pci device listed (there sh=
=3D
ould be many listed).
I checked first:=
t=3D
he IOMMU setting is set in the kernel.
<=3D
br data-mce-bogus=3D3D"1">Our environment:<=3D
br data-mce-bogus=3D3D"1">
We have two HV running CentOs7.
<=
di=3D
v>Each of them have three fiber channel cards. First card is connected to a=
=3D
disk array, second one to the first tape library and third one to another =
=3D
tape library.
We used to give the second card to a VM1, and t=
=3D
he third card to another VM2. Both VMs ran on one HV or on the other in the=
=3D
past: everything was fine.
What happened:
Today we add=
t=3D
o upgrade one of the HV (HV1). Both VMs ran on this HV1. We upgraded then r=
=3D
estarted the HV1.
Then I edited the se=
co=3D
nd VM2 in order to remove card from HV1. Then I tried to add the card from =
=3D
HV2 to this VM2: the list of "host devices" is empty.
I started the VM1: it works fine (I didn't change anything on=
=3D
it).
--
![](3D3D"https://www.cines.fr/wp-content/uploads/2014/01/logo=)
<=
/td>
|
| &=
nb=3D
sp; |
J=3DC3=3DA9r=3DC3=3DB4me Chapelle |
| &=
nb=3D
sp; |
Centre Informatique
National de l'<=
=3D
b>Enseignement
Sup=3DC3=3DA9rieur<=3D
br>
950 rue de Saint Priest
34097 MONTPELLIER
Cedex 5
|
|
| T=3DC3=3DA9l. |
04 67 14 14 14 |
| |
eMai=
l<=3D
/td>
| chapelle(a)cines.fr
|
|
|
=3D20
------=3D_Part_366645_1002603632.1465311035701--
--===============6350193133577545074==
Content-Type: multipart/alternative
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.bin"
LS0tLS0tPV9QYXJ0XzM2NjY0NV8xMDAyNjAzNjMyLjE0NjUzMTEwMzU3MDEKQ29udGVudC1UeXBl
OiB0ZXh0L3BsYWluOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1
b3RlZC1wcmludGFibGUKCldlIGFyZSBubyBtb3JlIGFibGUgdG8gZ2l2ZSBhIHBjaSBkZXZpY2Ug
dG8gb25lIFZNLj0yMApIZXJlIGlzIHRoZSBwcm9jZXNzIHdlIHVzZWQgdG8gYXNzaWduIG9uZSBk
ZXZpY2UgdG8gb25lIFZNOj0yMAotIENsaWNrIG9uIHRoZSBWTSwgYW5kIHRoZW4gb24gdGhlIHRh
YiAiSG9zdCBkZXZpY2VzIj0yMAotIEluIHRoZSB0YWIsIGNsaWNrIG9uICJwaW4gdG8gaG9zdCIg
YW5kIHNlbGVjdCB0aGUgaG9zdCB0aGF0IHdpbGwgZ2l2ZSB0aGU9CiBwY2kgZGV2aWNlPTIwCi0g
Q2xpY2sgb24gImFkZCBkZXZpY2UiLCBpbiB0aGUgd2luZG93IGFwcGVhciBhbGwgdGhlIHBjaSBk
ZXZpY2VzOiBjaG9vc2UgdD0KaGUgcGNpIGNhcmQgeW91IHdpc2ggKGV4LiBwY2lfMDAwMF84M18w
MF8wIHBjaV8wMDAwXzgzXzAwXzEpIGFuZCB0aGVuIGNsaWNrPQogb24gdGhlIGdyYXkgYXJyb3cg
dG8gYXNzaWduIHRoaXMgZGV2aWNlIHRvIHRoZSBWTT0yMAotIENsaWNrIG9uIG9rPTIwCi0gYm9v
dCB0aGUgVk0uPTIwClRoZSBwcm9ibGVtIGlzIHRoYXQgaW4gdGhlIHdpbmRvdyB0aGF0IGFwcGVh
cnMsIHRoZXJlIGlzIG5vdCBhIHNpbmdsZSBwY2kgZD0KZXZpY2UgbGlzdGVkICh0aGVyZSBzaG91
bGQgYmUgbWFueSBsaXN0ZWQpLj0yMApJIGNoZWNrZWQgZmlyc3Q6IHRoZSBJT01NVSBzZXR0aW5n
IGlzIHNldCBpbiB0aGUga2VybmVsLj0yMAoKT3VyIGVudmlyb25tZW50Oj0yMApXZSBoYXZlIHR3
byBIViBydW5uaW5nIENlbnRPczcuPTIwCkVhY2ggb2YgdGhlbSBoYXZlIHRocmVlIGZpYmVyIGNo
YW5uZWwgY2FyZHMuIEZpcnN0IGNhcmQgaXMgY29ubmVjdGVkIHRvIGEgZD0KaXNrIGFycmF5LCBz
ZWNvbmQgb25lIHRvIHRoZSBmaXJzdCB0YXBlIGxpYnJhcnkgYW5kIHRoaXJkIG9uZSB0byBhbm90
aGVyIHRhPQpwZSBsaWJyYXJ5Lj0yMApXZSB1c2VkIHRvIGdpdmUgdGhlIHNlY29uZCBjYXJkIHRv
IGEgVk0xLCBhbmQgdGhlIHRoaXJkIGNhcmQgdG8gYW5vdGhlciBWTTI9Ci4gQm90aCBWTXMgcmFu
IG9uIG9uZSBIViBvciBvbiB0aGUgb3RoZXIgaW4gdGhlIHBhc3Q6IGV2ZXJ5dGhpbmcgd2FzIGZp
bmUuPQo9MjAKV2hhdCBoYXBwZW5lZDo9MjAKVG9kYXkgd2UgYWRkIHRvIHVwZ3JhZGUgb25lIG9m
IHRoZSBIViAoSFYxKS4gQm90aCBWTXMgcmFuIG9uIHRoaXMgSFYxLiBXZSB1PQpwZ3JhZGVkIHRo
ZW4gcmVzdGFydGVkIHRoZSBIVjEuPTIwClRoZW4gSSBlZGl0ZWQgdGhlIHNlY29uZCBWTTIgaW4g
b3JkZXIgdG8gcmVtb3ZlIGNhcmQgZnJvbSBIVjEuIFRoZW4gSSB0cmllZD0KIHRvIGFkZCB0aGUg
Y2FyZCBmcm9tIEhWMiB0byB0aGlzIFZNMjogdGhlIGxpc3Qgb2YgImhvc3QgZGV2aWNlcyIgaXMg
ZW1wdHkuPQo9MjAKSSBzdGFydGVkIHRoZSBWTTE6IGl0IHdvcmtzIGZpbmUgKEkgZGlkbid0IGNo
YW5nZSBhbnl0aGluZyBvbiBpdCkuPTIwCgoKLS09MjAKCgo9MDkKPTA5Cj0wOQo9MDk9MDlKPUMz
PUE5cj1DMz1CNG1lIENoYXBlbGxlPTIwCj0wOT0wOUMgZW50cmUgSSBuZm9ybWF0aXF1ZSBOIGF0
aW9uYWwgZGUgbCcgRSBuc2VpZ25lbWVudCBTIHVwPUMzPUE5cmlldXI9Cj0yMAo5NTAgcnVlIGRl
IFNhaW50IFByaWVzdD0yMAozNCAwOTcgTSBPTlRQRUxMSUVSIENlZGV4IDU9MjAKCj0wOT0wOVQ9
QzM9QTlsLiA9MDkwNCA2NyAxNCAxNCAxND0yMAo9MDk9MDllTWFpbCA9MDljaGFwZWxsZUBjaW5l
cy5mcj0yMAoKCi0tLS0tLT1fUGFydF8zNjY2NDVfMTAwMjYwMzYzMi4xNDY1MzExMDM1NzAxCkNv
bnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5j
b2Rpbmc6IHF1b3RlZC1wcmludGFibGUKCjxodG1sPjxib2R5PjxkaXYgc3R5bGU9M0QiZm9udC1m
YW1pbHk6IFRpbWVzIE5ldyBSb21hbjsgZm9udC1zaXplOiAxMnB0OyBjbz0KbG9yOiAjMDAwMDAw
Ij48ZGl2PldlIGFyZSBubyBtb3JlIGFibGUgdG8gZ2l2ZSBhIHBjaSBkZXZpY2UgdG8gb25lIFZN
LjxiciBkPQphdGEtbWNlLWJvZ3VzPTNEIjEiPjwvZGl2PjxkaXY+SGVyZSBpcyB0aGUgcHJvY2Vz
cyB3ZSB1c2VkIHRvIGFzc2lnbiBvbmUgZGU9CnZpY2UgdG8gb25lIFZNOjxiciBkYXRhLW1jZS1i
b2d1cz0zRCIxIj48L2Rpdj48ZGl2Pi0gQ2xpY2sgb24gdGhlIFZNLCBhbmQgdD0KaGVuIG9uIHRo
ZSB0YWIgIkhvc3QgZGV2aWNlcyI8YnI+PC9kaXY+PGRpdj4tIEluIHRoZSB0YWIsIGNsaWNrIG9u
ICJwaW4gdG8gPQpob3N0IiBhbmQgc2VsZWN0IHRoZSBob3N0IHRoYXQgd2lsbCBnaXZlIHRoZSBw
Y2kgZGV2aWNlPGJyIGRhdGEtbWNlLWJvZ3VzPQo9M0QiMSI+PC9kaXY+PGRpdj4tIENsaWNrIG9u
ICJhZGQgZGV2aWNlIiwgaW4gdGhlIHdpbmRvdyBhcHBlYXIgYWxsIHRoZSBwY2k9CiBkZXZpY2Vz
OiBjaG9vc2UgdGhlIHBjaSBjYXJkIHlvdSB3aXNoIChleC4gcGNpXzAwMDBfODNfMDBfMCBwY2lf
MDAwMF84M18wMD0KXzEpIGFuZCB0aGVuIGNsaWNrIG9uIHRoZSBncmF5IGFycm93IHRvIGFzc2ln
biB0aGlzIGRldmljZSB0byB0aGUgVk08YnIgZGF0PQphLW1jZS1ib2d1cz0zRCIxIj48L2Rpdj48
ZGl2Pi0gQ2xpY2sgb24gb2s8YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGQ9Cml2Pi0g
Ym9vdCB0aGUgVk0uPGJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPjwvZGl2PjxkaXY+VGhlIHByb2Js
ZW0gaXMgdGhhdCBpbj0KIHRoZSB3aW5kb3cgdGhhdCBhcHBlYXJzLCB0aGVyZSBpcyBub3QgYSBz
aW5nbGUgcGNpIGRldmljZSBsaXN0ZWQgKHRoZXJlIHNoPQpvdWxkIGJlIG1hbnkgbGlzdGVkKS48
YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGRpdj5JIGNoZWNrZWQgZmlyc3Q6IHQ9Cmhl
IElPTU1VIHNldHRpbmcgaXMgc2V0IGluIHRoZSBrZXJuZWwuPGJyIGRhdGEtbWNlLWJvZ3VzPTNE
IjEiPjwvZGl2PjxkaXY+PD0KYnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGRpdj48c3Bh
biBkYXRhLW1jZS1zdHlsZT0zRCJ0ZXh0LWRlY29yYXRpb246PQogdW5kZXJsaW5lOyIgc3R5bGU9
M0QidGV4dC1kZWNvcmF0aW9uOiB1bmRlcmxpbmU7Ij5PdXIgZW52aXJvbm1lbnQ6PC9zcGFuPjw9
CmJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPjwvZGl2PjxkaXY+V2UgaGF2ZSB0d28gSFYgcnVubmlu
ZyBDZW50T3M3LjwvZGl2PjxkaT0Kdj5FYWNoIG9mIHRoZW0gaGF2ZSB0aHJlZSBmaWJlciBjaGFu
bmVsIGNhcmRzLiBGaXJzdCBjYXJkIGlzIGNvbm5lY3RlZCB0byBhPQogZGlzayBhcnJheSwgc2Vj
b25kIG9uZSB0byB0aGUgZmlyc3QgdGFwZSBsaWJyYXJ5IGFuZCB0aGlyZCBvbmUgdG8gYW5vdGhl
ciA9CnRhcGUgbGlicmFyeS48YnI+PC9kaXY+PGRpdj5XZSB1c2VkIHRvIGdpdmUgdGhlIHNlY29u
ZCBjYXJkIHRvIGEgVk0xLCBhbmQgdD0KaGUgdGhpcmQgY2FyZCB0byBhbm90aGVyIFZNMi4gQm90
aCBWTXMgcmFuIG9uIG9uZSBIViBvciBvbiB0aGUgb3RoZXIgaW4gdGhlPQogcGFzdDogZXZlcnl0
aGluZyB3YXMgZmluZS48YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGRpdj48c3BhbiBk
YXRhLW09CmNlLXN0eWxlPTNEInRleHQtZGVjb3JhdGlvbjogdW5kZXJsaW5lOyIgc3R5bGU9M0Qi
dGV4dC1kZWNvcmF0aW9uOiB1bmRlcmxpbj0KZTsiPldoYXQgaGFwcGVuZWQ6PC9zcGFuPjxiciBk
YXRhLW1jZS1ib2d1cz0zRCIxIj48L2Rpdj48ZGl2PlRvZGF5IHdlIGFkZCB0PQpvIHVwZ3JhZGUg
b25lIG9mIHRoZSBIViAoSFYxKS4gQm90aCBWTXMgcmFuIG9uIHRoaXMgSFYxLiBXZSB1cGdyYWRl
ZCB0aGVuIHI9CmVzdGFydGVkIHRoZSBIVjEuPGJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPjwvZGl2
PjxkaXY+VGhlbiBJIGVkaXRlZCB0aGUgc2Vjbz0KbmQgVk0yIGluIG9yZGVyIHRvIHJlbW92ZSBj
YXJkIGZyb20gSFYxLiBUaGVuIEkgdHJpZWQgdG8gYWRkIHRoZSBjYXJkIGZyb20gPQpIVjIgdG8g
dGhpcyBWTTI6IHRoZSBsaXN0IG9mICJob3N0IGRldmljZXMiIGlzIGVtcHR5LjxiciBkYXRhLW1j
ZS1ib2d1cz0zRCI9CjEiPjwvZGl2PjxkaXY+SSBzdGFydGVkIHRoZSBWTTE6IGl0IHdvcmtzIGZp
bmUgKEkgZGlkbid0IGNoYW5nZSBhbnl0aGluZyBvbj0KIGl0KS48YnIgZGF0YS1tY2UtYm9ndXM9
M0QiMSI+PC9kaXY+PGRpdj48YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGRpPQp2Pjxi
cj48L2Rpdj48ZGl2IGRhdGEtbWFya2VyPTNEIl9fU0lHX1BPU1RfXyI+LS0gPGJyPjwvZGl2Pjxk
aXY+PGJyPgogICAgICA8dGFibGUgc3R5bGU9M0QiYm9yZGVyOiAycHggc29saWQiPgogICAgICAg
IDx0Ym9keT4KICAgICAgICAgIDx0cj4KICAgICAgICAgICAgPHRkPgogICAgICAgICAgICAgIDx0
YWJsZSBjZWxscGFkZGluZz0zRCIwIiBjZWxsc3BhY2luZz0zRCIwIiBib3JkZXI9M0QiMCI+CiAg
ICAgICAgICAgICAgICA8dGJvZHk+CiAgICAgICAgICAgICAgICAgIDx0cj4KICAgICAgICAgICAg
ICAgICAgICA8dGQ+PGEgaHJlZj0zRCJodHRwOi8vd3d3LmNpbmVzLmZyIj48aW1nIHN0eWxlPTNE
ImJvcmRlPQpyOiAwcHggbm9uZTsiIHNyYz0zRCJodHRwczovL3d3dy5jaW5lcy5mci93cC1jb250
ZW50L3VwbG9hZHMvMjAxNC8wMS9sb2dvLXM9Cml0ZTIxMS5wbmciIGhlaWdodD0zRCI0MyIgd2lk
dGg9M0QiNjAiPjwvYT48YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC90ZD4KICAgICAgICAgICAg
ICAgICAgICA8dGQ+CiAgICAgICAgICAgICAgICAgICAgICA8dGFibGUgY2VsbHBhZGRpbmc9M0Qi
MCIgY2VsbHNwYWNpbmc9M0QiMCI+CiAgICAgICAgICAgICAgICAgICAgICAgIDx0Ym9keT4KICAg
ICAgICAgICAgICAgICAgICAgICAgICA8dHI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
dGQgc3R5bGU9M0QiYm9yZGVyLWJvdHRvbTogMXB4IHNvbGlkCiAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIDtib3JkZXItbGVmdDogMXB4IHNvbGlkIDsgcGFkZGluZy10b3A6CiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIDJweDsgcGFkZGluZy1ib3R0b206IDJweDsiIHdpZHRoPTNE
IjMycHgiPiZuYj0Kc3A7PC90ZD4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBzdHls
ZT0zRCJib3JkZXItYm90dG9tOiAxcHggc29saWQgOyIgY29sc3BhPQpuPTNEIjIiPjxiaWc+Sj1D
Mz1BOXI9QzM9QjRtZSBDaGFwZWxsZTwvYmlnPjwvdGQ+CiAgICAgICAgICAgICAgICAgICAgICAg
ICAgPC90cj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8dHI+CiAgICAgICAgICAgICAgICAg
ICAgICAgICAgICA8dGQgc3R5bGU9M0QiYm9yZGVyLWJvdHRvbTogMXB4IHNvbGlkCiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIDtib3JkZXItbGVmdDogMXB4IHNvbGlkIDsgcGFkZGluZy10
b3A6CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDJweDsgcGFkZGluZy1ib3R0b206IDJw
eDsiIHdpZHRoPTNEIjMycHgiPiZuYj0Kc3A7PC90ZD4KICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIDx0ZCBzdHlsZT0zRCJib3JkZXItYm90dG9tOiAxcHggc29saWQgOyIgY29sc3BhPQpuPTNE
IjIiPiA8Yj5DPC9iPjxzbWFsbD5lbnRyZSA8L3NtYWxsPjxiPkk8L2I+PHNtYWxsPm5mb3JtYXRp
cXVlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvc21hbGw+PGI+TjwvYj48c21hbGw+
YXRpb25hbCBkZSBsJzwvc21hbGw+PD0KYj5FPC9iPjxzbWFsbD5uc2VpZ25lbWVudAogICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICA8L3NtYWxsPjxiPlM8L2I+PHNtYWxsPnVwPUMzPUE5cmll
dXI8L3NtYWxsPjw9CmJyPgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8c21hbGw+OTUw
IHJ1ZSBkZSBTYWludCBQcmllc3Q8L3NtYWxsPjxicj4KICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgMzQ8c21hbGw+MDk3PC9zbWFsbD4gTTxzbWFsbD5PTlRQRUxMSUVSCiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgQ2VkZXggNTwvc21hbGw+PGJyPgogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgPC90ZD4KICAgICAgICAgICAgICAgICAgICAgICAgICA8L3RyPgogICAgICAg
ICAgICAgICAgICAgICAgICAgIDx0cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBz
dHlsZT0zRCJib3JkZXItbGVmdDogMXB4IHNvbGlkIDsKICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgcGFkZGluZy10b3A6IDJweDsgIiB3aWR0aD0zRCIzMnB4Ij4mbmJzcDs8L3RkPQo+CiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICA8dGQ+VD1DMz1BOWwuPC90ZD4KICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIDx0ZD4wNCA2NyAxNCAxNCAxNDwvdGQ+CiAgICAgICAgICAgICAgICAg
ICAgICAgICAgPC90cj4KICAgICAgICAgICAgICAgICAgICAgICAgICA8dHI+CiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICA8dGQgc3R5bGU9M0QiYm9yZGVyLWJvdHRvbTogMXB4IHNvbGlkCiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDtib3JkZXItbGVmdDogMXB4IHNvbGlkIDsgcGFk
ZGluZy1ib3R0b206CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDJweDsiIHdpZHRoPTNE
IjMycHgiPiZuYnNwOzwvdGQ+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICA8dGQgc3R5bGU9
M0QiYm9yZGVyLWJvdHRvbTogMXB4IHNvbGlkIDsiPmVNYWlsPD0KL3RkPgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPHRkIHN0eWxlPTNEImJvcmRlci1ib3R0b206IDFweCBzb2xpZCAiPjxh
IGhyZWY9Cj0zRCJtYWlsdG86Y2hhcGVsbGVAY2luZXMuZnIiPmNoYXBlbGxlQGNpbmVzLmZyPC9h
PjxiciBkYXRhLW1jZS1ib2d1cz0zRCIxIj0KPjwvdGQ+CiAgICAgICAgICAgICAgICAgICAgICAg
ICAgPC90cj4KICAgICAgICAgICAgICAgICAgICAgICAgPC90Ym9keT4KICAgICAgICAgICAgICAg
ICAgICAgIDwvdGFibGU+CiAgICAgICAgICAgICAgICAgICAgPC90ZD4KICAgICAgICAgICAgICAg
ICAgPC90cj4KICAgICAgICAgICAgICAgIDwvdGJvZHk+CiAgICAgICAgICAgICAgPC90YWJsZT4K
ICAgICAgICAgICAgPC90ZD4KICAgICAgICAgIDwvdHI+CiAgICAgICAgPC90Ym9keT4KICAgICAg
PC90YWJsZT4KICAgPTIwCiAgICA8YnI+PC9kaXY+PC9kaXY+PC9ib2R5PjwvaHRtbD4KLS0tLS0t
PV9QYXJ0XzM2NjY0NV8xMDAyNjAzNjMyLjE0NjUzMTEwMzU3MDEtLQo=
--===============6350193133577545074==--
From mpolednik at redhat.com Tue Jun 7 11:11:13 2016
Content-Type: multipart/mixed; boundary="===============3451777252430136115=="
MIME-Version: 1.0
From: Martin Polednik
To: users at ovirt.org
Subject: Re: [ovirt-users] PCI Passthrough problem
Date: Tue, 07 Jun 2016 17:11:10 +0200
Message-ID: <20160607151107.GB32730@Alexandra.local>
In-Reply-To: 244701585.366646.1465311035702.JavaMail.zimbra@cines.fr
--===============3451777252430136115==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
On 07/06/16 16:50 +0200, J=C3=A9r=C3=B4me Chapelle wrote:
>We are no more able to give a pci device to one VM.
>Here is the process we used to assign one device to one VM:
>- Click on the VM, and then on the tab "Host devices"
>- In the tab, click on "pin to host" and select the host that will give th=
e pci device
>- Click on "add device", in the window appear all the pci devices: choose =
the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then clic=
k on the gray arrow to assign this device to the VM
>- Click on ok
>- boot the VM.
>The problem is that in the window that appears, there is not a single pci =
device listed (there should be many listed).
>I checked first: the IOMMU setting is set in the kernel.
>
>Our environment:
>We have two HV running CentOs7.
>Each of them have three fiber channel cards. First card is connected to a =
disk array, second one to the first tape library and third one to another t=
ape library.
>We used to give the second card to a VM1, and the third card to another VM=
2. Both VMs ran on one HV or on the other in the past: everything was fine.
>What happened:
>Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. We =
upgraded then restarted the HV1.
>Then I edited the second VM2 in order to remove card from HV1. Then I trie=
d to add the card from HV2 to this VM2: the list of "host devices" is empty.
>I started the VM1: it works fine (I didn't change anything on it).
Hello!
I'm afraid you have hit the bug [1]. For hosts where you need host
devices, the best solution is most likely not upgrading them to 3.6.6
for now.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1341299
>
>-- =
>
>
> =
> =
> =
> J=C3=A9r=C3=B4me Chapelle
> C entre I nformatique N ational de l' E nseignement S up=C3=A9rieur
>950 rue de Saint Priest
>34 097 M ONTPELLIER Cedex 5
>
> T=C3=A9l. 04 67 14 14 14
> eMail chapelle(a)cines.fr
>
>_______________________________________________
>Users mailing list
>Users(a)ovirt.org
>http://lists.ovirt.org/mailman/listinfo/users
--===============3451777252430136115==--
From caignec at cines.fr Tue Jun 7 13:14:04 2016
Content-Type: multipart/mixed; boundary="===============1128010530933901794=="
MIME-Version: 1.0
From: Lionel Caignec
To: users at ovirt.org
Subject: Re: [ovirt-users] PCI Passthrough problem
Date: Tue, 07 Jun 2016 19:04:14 +0200
Message-ID: <1815494073.370341.1465319054024.JavaMail.zimbra@cines.fr>
In-Reply-To: 20160607151107.GB32730@Alexandra.local
--===============1128010530933901794==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Hi i'm coworker of Mr Chapelle,
Thank you for your reply.
so if i understand we need to "downgrade" our ovirt engine which is in vers=
ion 3.6.6.x to version 3.6.5?
But how can we do that?
Lionel Caignec.
----- Mail original -----
De: "Martin Polednik"
=C3=80: "chapelle"
Cc: "users" , "honvault"
Envoy=C3=A9: Mardi 7 Juin 2016 17:11:10
Objet: Re: [ovirt-users] PCI Passthrough problem
On 07/06/16 16:50 +0200, J=C3=A9r=C3=B4me Chapelle wrote:
>We are no more able to give a pci device to one VM.
>Here is the process we used to assign one device to one VM:
>- Click on the VM, and then on the tab "Host devices"
>- In the tab, click on "pin to host" and select the host that will give th=
e pci device
>- Click on "add device", in the window appear all the pci devices: choose =
the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then clic=
k on the gray arrow to assign this device to the VM
>- Click on ok
>- boot the VM.
>The problem is that in the window that appears, there is not a single pci =
device listed (there should be many listed).
>I checked first: the IOMMU setting is set in the kernel.
>
>Our environment:
>We have two HV running CentOs7.
>Each of them have three fiber channel cards. First card is connected to a =
disk array, second one to the first tape library and third one to another t=
ape library.
>We used to give the second card to a VM1, and the third card to another VM=
2. Both VMs ran on one HV or on the other in the past: everything was fine.
>What happened:
>Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. We =
upgraded then restarted the HV1.
>Then I edited the second VM2 in order to remove card from HV1. Then I trie=
d to add the card from HV2 to this VM2: the list of "host devices" is empty.
>I started the VM1: it works fine (I didn't change anything on it).
Hello!
I'm afraid you have hit the bug [1]. For hosts where you need host
devices, the best solution is most likely not upgrading them to 3.6.6
for now.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1341299
>
>-- =
>
>
> =
> =
> =
> J=C3=A9r=C3=B4me Chapelle
> C entre I nformatique N ational de l' E nseignement S up=C3=A9rieur
>950 rue de Saint Priest
>34 097 M ONTPELLIER Cedex 5
>
> T=C3=A9l. 04 67 14 14 14
> eMail chapelle(a)cines.fr
>
>_______________________________________________
>Users mailing list
>Users(a)ovirt.org
>http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--===============1128010530933901794==--
From mpolednik at redhat.com Wed Jun 8 02:20:07 2016
Content-Type: multipart/mixed; boundary="===============4687531214226309279=="
MIME-Version: 1.0
From: Martin Polednik
To: users at ovirt.org
Subject: Re: [ovirt-users] PCI Passthrough problem
Date: Wed, 08 Jun 2016 08:20:02 +0200
Message-ID: <20160608062001.GA1220@dhcp131-83.brq.redhat.com>
In-Reply-To: 1815494073.370341.1465319054024.JavaMail.zimbra@cines.fr
--===============4687531214226309279==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
On 07/06/16 19:04 +0200, Lionel Caignec wrote:
>Hi i'm coworker of Mr Chapelle,
>
>Thank you for your reply.
>
>so if i understand we need to "downgrade" our ovirt engine which is in ver=
sion 3.6.6.x to version 3.6.5?
>But how can we do that?
You would only need to downgrade the 2 hosts to 3.6.5 VDSM. But
thinking of it, you could instead upgrade to 3.6.7 (rc2) - the bug is
fixed there.
>Lionel Caignec.
>
>
>
>----- Mail original -----
>De: "Martin Polednik"
>=C3=80: "chapelle"
>Cc: "users" , "honvault"
>Envoy=C3=A9: Mardi 7 Juin 2016 17:11:10
>Objet: Re: [ovirt-users] PCI Passthrough problem
>
>On 07/06/16 16:50 +0200, J=C3=A9r=C3=B4me Chapelle wrote:
>>We are no more able to give a pci device to one VM.
>>Here is the process we used to assign one device to one VM:
>>- Click on the VM, and then on the tab "Host devices"
>>- In the tab, click on "pin to host" and select the host that will give t=
he pci device
>>- Click on "add device", in the window appear all the pci devices: choose=
the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then cli=
ck on the gray arrow to assign this device to the VM
>>- Click on ok
>>- boot the VM.
>>The problem is that in the window that appears, there is not a single pci=
device listed (there should be many listed).
>>I checked first: the IOMMU setting is set in the kernel.
>>
>>Our environment:
>>We have two HV running CentOs7.
>>Each of them have three fiber channel cards. First card is connected to a=
disk array, second one to the first tape library and third one to another =
tape library.
>>We used to give the second card to a VM1, and the third card to another V=
M2. Both VMs ran on one HV or on the other in the past: everything was fine.
>>What happened:
>>Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. We=
upgraded then restarted the HV1.
>>Then I edited the second VM2 in order to remove card from HV1. Then I tri=
ed to add the card from HV2 to this VM2: the list of "host devices" is empt=
y.
>>I started the VM1: it works fine (I didn't change anything on it).
>
>Hello!
>
>I'm afraid you have hit the bug [1]. For hosts where you need host
>devices, the best solution is most likely not upgrading them to 3.6.6
>for now.
>
>[1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1341299
>
>>
>>--
>>
>>
>> =
>> =
>> =
>> J=C3=A9r=C3=B4me Chapelle
>> C entre I nformatique N ational de l' E nseignement S up=C3=A9rieur
>>950 rue de Saint Priest
>>34 097 M ONTPELLIER Cedex 5
>>
>> T=C3=A9l. 04 67 14 14 14
>> eMail chapelle(a)cines.fr
>>
>
>>_______________________________________________
>>Users mailing list
>>Users(a)ovirt.org
>>http://lists.ovirt.org/mailman/listinfo/users
>
>_______________________________________________
>Users mailing list
>Users(a)ovirt.org
>http://lists.ovirt.org/mailman/listinfo/users
--===============4687531214226309279==--
From caignec at cines.fr Wed Jun 8 02:41:46 2016
Content-Type: multipart/mixed; boundary="===============7451552750412408436=="
MIME-Version: 1.0
From: Lionel Caignec
To: users at ovirt.org
Subject: Re: [ovirt-users] PCI Passthrough problem
Date: Wed, 08 Jun 2016 08:41:42 +0200
Message-ID: <702501160.374366.1465368102312.JavaMail.zimbra@cines.fr>
In-Reply-To: 20160608062001.GA1220@dhcp131-83.brq.redhat.com
--===============7451552750412408436==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Hi thanks for reply
There is something i does not understand, on my hosts i've vdsm 4.17.28-0.e=
l7 no indication about a version 3.6.5 or 3.6.7.
How can i get this version?
----- Mail original -----
De: "Martin Polednik"
=C3=80: "Lionel Caignec"
Cc: "Martin Polednik" , "chapelle" , "users" , "honvault"
Envoy=C3=A9: Mercredi 8 Juin 2016 08:20:02
Objet: Re: PCI Passthrough problem
On 07/06/16 19:04 +0200, Lionel Caignec wrote:
>Hi i'm coworker of Mr Chapelle,
>
>Thank you for your reply.
>
>so if i understand we need to "downgrade" our ovirt engine which is in ver=
sion 3.6.6.x to version 3.6.5?
>But how can we do that?
You would only need to downgrade the 2 hosts to 3.6.5 VDSM. But
thinking of it, you could instead upgrade to 3.6.7 (rc2) - the bug is
fixed there.
>Lionel Caignec.
>
>
>
>----- Mail original -----
>De: "Martin Polednik"
>=C3=80: "chapelle"
>Cc: "users" , "honvault"
>Envoy=C3=A9: Mardi 7 Juin 2016 17:11:10
>Objet: Re: [ovirt-users] PCI Passthrough problem
>
>On 07/06/16 16:50 +0200, J=C3=A9r=C3=B4me Chapelle wrote:
>>We are no more able to give a pci device to one VM.
>>Here is the process we used to assign one device to one VM:
>>- Click on the VM, and then on the tab "Host devices"
>>- In the tab, click on "pin to host" and select the host that will give t=
he pci device
>>- Click on "add device", in the window appear all the pci devices: choose=
the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then cli=
ck on the gray arrow to assign this device to the VM
>>- Click on ok
>>- boot the VM.
>>The problem is that in the window that appears, there is not a single pci=
device listed (there should be many listed).
>>I checked first: the IOMMU setting is set in the kernel.
>>
>>Our environment:
>>We have two HV running CentOs7.
>>Each of them have three fiber channel cards. First card is connected to a=
disk array, second one to the first tape library and third one to another =
tape library.
>>We used to give the second card to a VM1, and the third card to another V=
M2. Both VMs ran on one HV or on the other in the past: everything was fine.
>>What happened:
>>Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. We=
upgraded then restarted the HV1.
>>Then I edited the second VM2 in order to remove card from HV1. Then I tri=
ed to add the card from HV2 to this VM2: the list of "host devices" is empt=
y.
>>I started the VM1: it works fine (I didn't change anything on it).
>
>Hello!
>
>I'm afraid you have hit the bug [1]. For hosts where you need host
>devices, the best solution is most likely not upgrading them to 3.6.6
>for now.
>
>[1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1341299
>
>>
>>--
>>
>>
>> =
>> =
>> =
>> J=C3=A9r=C3=B4me Chapelle
>> C entre I nformatique N ational de l' E nseignement S up=C3=A9rieur
>>950 rue de Saint Priest
>>34 097 M ONTPELLIER Cedex 5
>>
>> T=C3=A9l. 04 67 14 14 14
>> eMail chapelle(a)cines.fr
>>
>
>>_______________________________________________
>>Users mailing list
>>Users(a)ovirt.org
>>http://lists.ovirt.org/mailman/listinfo/users
>
>_______________________________________________
>Users mailing list
>Users(a)ovirt.org
>http://lists.ovirt.org/mailman/listinfo/users
--===============7451552750412408436==--
From michal.skrivanek at redhat.com Wed Jun 8 03:12:47 2016
Content-Type: multipart/mixed; boundary="===============0364721818082211532=="
MIME-Version: 1.0
From: Michal Skrivanek
To: users at ovirt.org
Subject: Re: [ovirt-users] PCI Passthrough problem
Date: Wed, 08 Jun 2016 09:12:44 +0200
Message-ID: <09890770-6481-4C6F-91DB-2558FA123146@redhat.com>
In-Reply-To: 702501160.374366.1465368102312.JavaMail.zimbra@cines.fr
--===============0364721818082211532==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
> On 08 Jun 2016, at 08:41, Lionel Caignec wrote:
> =
> Hi thanks for reply
> =
> There is something i does not understand, on my hosts i've vdsm 4.17.28-0=
.el7 no indication about a version 3.6.5 or 3.6.7.
> How can i get this version?
The ovirt release 3.6.7 rc2, see Sandro=E2=80=99s announcement email sent t=
o this list last Thursday. Once you reconfigure repos as explained there yo=
u will see the new fixed vdsm:) It=E2=80=99s a release candidate repo
Hope it works,
michal
> =
> =
> =
> =
> ----- Mail original -----
> De: "Martin Polednik"
> =C3=80: "Lionel Caignec"
> Cc: "Martin Polednik" , "chapelle" , "users" , "honvault"
> Envoy=C3=A9: Mercredi 8 Juin 2016 08:20:02
> Objet: Re: PCI Passthrough problem
> =
> On 07/06/16 19:04 +0200, Lionel Caignec wrote:
>> Hi i'm coworker of Mr Chapelle,
>> =
>> Thank you for your reply.
>> =
>> so if i understand we need to "downgrade" our ovirt engine which is in v=
ersion 3.6.6.x to version 3.6.5?
>> But how can we do that?
> =
> You would only need to downgrade the 2 hosts to 3.6.5 VDSM. But
> thinking of it, you could instead upgrade to 3.6.7 (rc2) - the bug is
> fixed there.
> =
>> Lionel Caignec.
>> =
>> =
>> =
>> ----- Mail original -----
>> De: "Martin Polednik"
>> =C3=80: "chapelle"
>> Cc: "users" , "honvault"
>> Envoy=C3=A9: Mardi 7 Juin 2016 17:11:10
>> Objet: Re: [ovirt-users] PCI Passthrough problem
>> =
>> On 07/06/16 16:50 +0200, J=C3=A9r=C3=B4me Chapelle wrote:
>>> We are no more able to give a pci device to one VM.
>>> Here is the process we used to assign one device to one VM:
>>> - Click on the VM, and then on the tab "Host devices"
>>> - In the tab, click on "pin to host" and select the host that will give=
the pci device
>>> - Click on "add device", in the window appear all the pci devices: choo=
se the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then c=
lick on the gray arrow to assign this device to the VM
>>> - Click on ok
>>> - boot the VM.
>>> The problem is that in the window that appears, there is not a single p=
ci device listed (there should be many listed).
>>> I checked first: the IOMMU setting is set in the kernel.
>>> =
>>> Our environment:
>>> We have two HV running CentOs7.
>>> Each of them have three fiber channel cards. First card is connected to=
a disk array, second one to the first tape library and third one to anothe=
r tape library.
>>> We used to give the second card to a VM1, and the third card to another=
VM2. Both VMs ran on one HV or on the other in the past: everything was fi=
ne.
>>> What happened:
>>> Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. =
We upgraded then restarted the HV1.
>>> Then I edited the second VM2 in order to remove card from HV1. Then I t=
ried to add the card from HV2 to this VM2: the list of "host devices" is em=
pty.
>>> I started the VM1: it works fine (I didn't change anything on it).
>> =
>> Hello!
>> =
>> I'm afraid you have hit the bug [1]. For hosts where you need host
>> devices, the best solution is most likely not upgrading them to 3.6.6
>> for now.
>> =
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1341299
>> =
>>> =
>>> --
>>> =
>>> =
>>> =
>>> =
>>> =
>>> J=C3=A9r=C3=B4me Chapelle
>>> C entre I nformatique N ational de l' E nseignement S up=C3=A9rieur
>>> 950 rue de Saint Priest
>>> 34 097 M ONTPELLIER Cedex 5
>>> =
>>> T=C3=A9l. 04 67 14 14 14
>>> eMail chapelle(a)cines.fr
>>> =
>> =
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>> =
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> =
> =
--===============0364721818082211532==--
From chapelle at cines.fr Wed Jun 8 04:24:43 2016
Content-Type: multipart/mixed; boundary="===============0514765152618994760=="
MIME-Version: 1.0
From: =?utf-8?q?J=C3=A9r=C3=B4me_Chapelle_=3Cchapelle_at_cines=2Efr=3E?=
To: users at ovirt.org
Subject: Re: [ovirt-users] PCI Passthrough problem
Date: Wed, 08 Jun 2016 10:24:36 +0200
Message-ID: <2145943052.378525.1465374276867.JavaMail.zimbra@cines.fr>
In-Reply-To: 09890770-6481-4C6F-91DB-2558FA123146@redhat.com
--===============0514765152618994760==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
------=3D_Part_378524_1670661841.1465374276865
Content-Type: text/plain; charset=3Dutf-8
Content-Transfer-Encoding: quoted-printable
Thanks to all of us !=3D20
We read the Sandro's email annoucement:=3D20
http://lists.ovirt.org/pipermail/users/2016-June/040172.html=3D20
And applied to the HV the update process pointed in:=3D20
http://www.ovirt.org/release/3.6.7/=3D20
(we only updated vdsm)=3D20
Now our HV is working fine and we are able again to do pci passthrough with=
=3D
it.=3D20
As a result, the virtual machine hosted on it now sees tape.=3D20
Thanks again for your fast and precisous help !=3D20
Jerome=3D20
From: "Michal Skrivanek" =3D20
To: "Lionel Caignec" =3D20
Cc: "honvault" , "Martin Polednik" , "users" =3D20
Sent: Wednesday, June 8, 2016 9:12:44 AM=3D20
Subject: Re: [ovirt-users] PCI Passthrough problem=3D20
> On 08 Jun 2016, at 08:41, Lionel Caignec wrote:=3D20
>=3D20
> Hi thanks for reply=3D20
>=3D20
> There is something i does not understand, on my hosts i've vdsm 4.17.28-0=
=3D
.el7 no indication about a version 3.6.5 or 3.6.7.=3D20
> How can i get this version?=3D20
The ovirt release 3.6.7 rc2, see Sandro=3DE2=3D80=3D99s announcement email =
sent t=3D
o this list last Thursday. Once you reconfigure repos as explained there yo=
=3D
u will see the new fixed vdsm:) It=3DE2=3D80=3D99s a release candidate repo=
=3D20
Hope it works,=3D20
michal=3D20
>=3D20
>=3D20
>=3D20
>=3D20
> ----- Mail original -----=3D20
> De: "Martin Polednik" =3D20
> =3DC3=3D80: "Lionel Caignec" =3D20
> Cc: "Martin Polednik" , "chapelle" , "users" , "honvault" =3D20
> Envoy=3DC3=3DA9: Mercredi 8 Juin 2016 08:20:02=3D20
> Objet: Re: PCI Passthrough problem=3D20
>=3D20
> On 07/06/16 19:04 +0200, Lionel Caignec wrote:=3D20
>> Hi i'm coworker of Mr Chapelle,=3D20
>>=3D20
>> Thank you for your reply.=3D20
>>=3D20
>> so if i understand we need to "downgrade" our ovirt engine which is in v=
=3D
ersion 3.6.6.x to version 3.6.5?=3D20
>> But how can we do that?=3D20
>=3D20
> You would only need to downgrade the 2 hosts to 3.6.5 VDSM. But=3D20
> thinking of it, you could instead upgrade to 3.6.7 (rc2) - the bug is=3D20
> fixed there.=3D20
>=3D20
>> Lionel Caignec.=3D20
>>=3D20
>>=3D20
>>=3D20
>> ----- Mail original -----=3D20
>> De: "Martin Polednik" =3D20
>> =3DC3=3D80: "chapelle" =3D20
>> Cc: "users" , "honvault" =3D20
>> Envoy=3DC3=3DA9: Mardi 7 Juin 2016 17:11:10=3D20
>> Objet: Re: [ovirt-users] PCI Passthrough problem=3D20
>>=3D20
>> On 07/06/16 16:50 +0200, J=3DC3=3DA9r=3DC3=3DB4me Chapelle wrote:=3D20
>>> We are no more able to give a pci device to one VM.=3D20
>>> Here is the process we used to assign one device to one VM:=3D20
>>> - Click on the VM, and then on the tab "Host devices"=3D20
>>> - In the tab, click on "pin to host" and select the host that will give=
=3D
the pci device=3D20
>>> - Click on "add device", in the window appear all the pci devices: choo=
=3D
se the pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then c=
=3D
lick on the gray arrow to assign this device to the VM=3D20
>>> - Click on ok=3D20
>>> - boot the VM.=3D20
>>> The problem is that in the window that appears, there is not a single p=
=3D
ci device listed (there should be many listed).=3D20
>>> I checked first: the IOMMU setting is set in the kernel.=3D20
>>>=3D20
>>> Our environment:=3D20
>>> We have two HV running CentOs7.=3D20
>>> Each of them have three fiber channel cards. First card is connected to=
=3D
a disk array, second one to the first tape library and third one to anothe=
=3D
r tape library.=3D20
>>> We used to give the second card to a VM1, and the third card to another=
=3D
VM2. Both VMs ran on one HV or on the other in the past: everything was fi=
=3D
ne.=3D20
>>> What happened:=3D20
>>> Today we add to upgrade one of the HV (HV1). Both VMs ran on this HV1. =
=3D
We upgraded then restarted the HV1.=3D20
>>> Then I edited the second VM2 in order to remove card from HV1. Then I t=
=3D
ried to add the card from HV2 to this VM2: the list of "host devices" is em=
=3D
pty.=3D20
>>> I started the VM1: it works fine (I didn't change anything on it).=3D20
>>=3D20
>> Hello!=3D20
>>=3D20
>> I'm afraid you have hit the bug [1]. For hosts where you need host=3D20
>> devices, the best solution is most likely not upgrading them to 3.6.6=3D=
20
>> for now.=3D20
>>=3D20
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=3D3D1341299=3D20
>>=3D20
>>>=3D20
>>> --=3D20
>>>=3D20
>>>=3D20
>>>=3D20
>>>=3D20
>>>=3D20
>>> J=3DC3=3DA9r=3DC3=3DB4me Chapelle=3D20
>>> C entre I nformatique N ational de l' E nseignement S up=3DC3=3DA9rieur=
=3D20
>>> 950 rue de Saint Priest=3D20
>>> 34 097 M ONTPELLIER Cedex 5=3D20
>>>=3D20
>>> T=3DC3=3DA9l. 04 67 14 14 14=3D20
>>> eMail chapelle(a)cines.fr=3D20
>>>=3D20
>>=3D20
>>> _______________________________________________=3D20
>>> Users mailing list=3D20
>>> Users(a)ovirt.org=3D20
>>> http://lists.ovirt.org/mailman/listinfo/users=3D20
>>=3D20
>> _______________________________________________=3D20
>> Users mailing list=3D20
>> Users(a)ovirt.org=3D20
>> http://lists.ovirt.org/mailman/listinfo/users=3D20
> _______________________________________________=3D20
> Users mailing list=3D20
> Users(a)ovirt.org=3D20
> http://lists.ovirt.org/mailman/listinfo/users=3D20
>=3D20
>=3D20
_______________________________________________=3D20
Users mailing list=3D20
Users(a)ovirt.org=3D20
http://lists.ovirt.org/mailman/listinfo/users=3D20
--=3D20
=3D09
=3D09
=3D09
=3D09=3D09J=3DC3=3DA9r=3DC3=3DB4me Chapelle=3D20
=3D09=3D09C entre I nformatique N ational de l' E nseignement S up=3DC3=3DA=
9rieur=3D
=3D20
950 rue de Saint Priest=3D20
34 097 M ONTPELLIER Cedex 5=3D20
=3D09=3D09T=3DC3=3DA9l. =3D0904 67 14 14 14=3D20
=3D09=3D09eMail =3D09chapelle(a)cines.fr=3D20
------=3D_Part_378524_1670661841.1465374276865
Content-Type: text/html; charset=3Dutf-8
Content-Transfer-Encoding: quoted-printable
Thanks to all of us !
We read the Sandro's=
=3D
email annoucement:
http://lists.ovirt=
.o=3D
rg/pipermail/users/2016-June/040172.html
And applied to the HV th=
=3D
e update process pointed in:
http://ww=
w.=3D
ovirt.org/release/3.6.7/
(we only updated vdsm)
Now our HV is working fine and we are able again to do =
pc=3D
i passthrough with it.
As a result, th=
e =3D
virtual machine hosted on it now sees tape.
<=3D
div>Thanks again for your fast and precisous help !
Jerome
F=
rom:=3D
"Michal Skrivanek" <michal.skrivanek(a)redhat.com>
To: "=3D
Lionel Caignec" <caignec(a)cines.fr>
Cc: "honvault" <hon=
va=3D
ult(a)cines.fr>, "Martin Polednik" <mpolednik(a)redhat.com>, "user=
s" &=3D
lt;users(a)ovirt.org>
Sent: Wednesday, June 8, 2016 9:12:44 AM=
Subject: Re: [ovirt-users] PCI Passthrough problem
> On 08 Jun 2016, at 08:41, Lionel =
Ca=3D
ignec <caignec(a)cines.fr> wrote:
>
> Hi thanks for repl=
y<=3D
br>>
> There is something i does not understand, on my hosts i've=
=3D
vdsm 4.17.28-0.el7 no indication about a version 3.6.5 or 3.6.7.
> H=
=3D
ow can i get this version?
The ovirt release 3.6.7 rc2, see Sandro=
=3D
=3DE2=3D80=3D99s announcement email sent to this list last Thursday. Once y=
ou rec=3D
onfigure repos as explained there you will see the new fixed vdsm:) It=3DE2=
=3D
=3D80=3D99s a release candidate repo
Hope it works,
michal
>=3D
;
>
>
>
> ----- Mail original -----
> De:=
=3D
"Martin Polednik" <mpolednik(a)redhat.com>
> =3DC3=3D80: "Lion=
el Cai=3D
gnec" <caignec(a)cines.fr>
> Cc: "Martin Polednik" <mpoledni=
k@=3D
redhat.com>, "chapelle" <chapelle(a)cines.fr>, "users" <users(a=
)ovi=3D
rt.org>, "honvault" <honvault(a)cines.fr>
> Envoy=3DC3=3DA9:=
Mercr=3D
edi 8 Juin 2016 08:20:02
> Objet: Re: PCI Passthrough problem
>=
=3D
> On 07/06/16 19:04 +0200, Lionel Caignec wrote:
>> Hi i'm=
=3D
coworker of Mr Chapelle,
>>
>> Thank you for your reply=
=3D
.
>>
>> so if i understand we need to "downgrade" our ov=
=3D
irt engine which is in version 3.6.6.x to version 3.6.5?
>> But ho=
=3D
w can we do that?
>
> You would only need to downgrade the 2 h=
=3D
osts to 3.6.5 VDSM. But
> thinking of it, you could instead upgrade t=
=3D
o 3.6.7 (rc2) - the bug is
> fixed there.
>
>> Lionel=
=3D
Caignec.
>>
>>
>>
>> ----- Mail ori=
=3D
ginal -----
>> De: "Martin Polednik" <mpolednik(a)redhat.com>=
;<=3D
br>>> =3DC3=3D80: "chapelle" <chapelle(a)cines.fr>
>> =
Cc: "u=3D
sers" <users(a)ovirt.org>, "honvault" <honvault(a)cines.fr>
=
>=3D
> Envoy=3DC3=3DA9: Mardi 7 Juin 2016 17:11:10
>> Objet: Re: [ov=
irt-=3D
users] PCI Passthrough problem
>>
>> On 07/06/16 16:50 +=
=3D
0200, J=3DC3=3DA9r=3DC3=3DB4me Chapelle wrote:
>>> We are no mo=
re able =3D
to give a pci device to one VM.
>>> Here is the process we used=
=3D
to assign one device to one VM:
>>> - Click on the VM, and the=
=3D
n on the tab "Host devices"
>>> - In the tab, click on "pin to =
=3D
host" and select the host that will give the pci device
>>> - C=
=3D
lick on "add device", in the window appear all the pci devices: choose the =
=3D
pci card you wish (ex. pci_0000_83_00_0 pci_0000_83_00_1) and then click on=
=3D
the gray arrow to assign this device to the VM
>>> - Click on =
=3D
ok
>>> - boot the VM.
>>> The problem is that in th=
=3D
e window that appears, there is not a single pci device listed (there shoul=
=3D
d be many listed).
>>> I checked first: the IOMMU setting is se=
=3D
t in the kernel.
>>>
>>> Our environment:
>&=
=3D
gt;> We have two HV running CentOs7.
>>> Each of them have t=
=3D
hree fiber channel cards. First card is connected to a disk array, second o=
=3D
ne to the first tape library and third one to another tape library.
>=
=3D
>> We used to give the second card to a VM1, and the third card to an=
=3D
other VM2. Both VMs ran on one HV or on the other in the past: everything w=
=3D
as fine.
>>> What happened:
>>> Today we add to upg=
=3D
rade one of the HV (HV1). Both VMs ran on this HV1. We upgraded then restar=
=3D
ted the HV1.
>>> Then I edited the second VM2 in order to remov=
=3D
e card from HV1. Then I tried to add the card from HV2 to this VM2: the lis=
=3D
t of "host devices" is empty.
>>> I started the VM1: it works f=
=3D
ine (I didn't change anything on it).
>>
>> Hello!
&g=
=3D
t;>
>> I'm afraid you have hit the bug [1]. For hosts where yo=
=3D
u need host
>> devices, the best solution is most likely not upgra=
=3D
ding them to 3.6.6
>> for now.
>>
>> [1] https:=
=3D
//bugzilla.redhat.com/show_bug.cgi?id=3D3D1341299
>>
>>&=
gt=3D
;
>>> --
>>>
>>>
>>> &nb=
=3D
sp;
>>> &=
=3D
nbsp;
>>> =
=3D
;
>>> &nb=
=3D
sp; J=3DC3=3DA9r=
=3DC3=3D
=3DB4me Chapelle
>>> =
&n=3D
bsp; C entre I nformatique N=
=3D
ational de l' E nseignement S up=3DC3=3DA9rieur
>>> 950 rue de=
Sai=3D
nt Priest
>>> 34 097 M ONTPELLIER Cedex 5
>>>
&=
=3D
gt;>> &nb=
=3D
sp; T=3DC3=3DA9l. &nbs=
p;&n=3D
bsp; 04 67 14 14 14
>>> &nbs=
=3D
p; eMail &=
=3D
nbsp; chapelle(a)cines.fr
>&=
gt=3D
;>
>>
>>> _______________________________________=
=3D
________
>>> Users mailing list
>>> Users(a)ovirt.o=
rg=3D
>>> http://lists.ovirt.org/mailman/listinfo/users
>> =
=3D
>> _______________________________________________
>> Us=
=3D
ers mailing list
>> Users(a)ovirt.org
>> http://lists.ovi=
rt=3D
.org/mailman/listinfo/users
> _______________________________________=
=3D
________
> Users mailing list
> Users(a)ovirt.org
> http:=
//=3D
lists.ovirt.org/mailman/listinfo/users
>
>
___________=
=3D
____________________________________
Users mailing list
Users(a)ovirt=
.o=3D
rg
http://lists.ovirt.org/mailman/listinfo/users
--
<=
/td>
|
| &=
nb=3D
sp; |
J=3DC3=3DA9r=3DC3=3DB4me Chapelle |
| &=
nb=3D
sp; |
Centre Informatique
National de l'<=
=3D
b>Enseignement
Sup=3DC3=3DA9rieur<=3D
br>
950 rue de Saint Priest
34097 MONTPELLIER
Cedex 5
|
|
| T=3DC3=3DA9l. |
04 67 14 14 14 |
| |
eMai=
l<=3D
/td>
| chapelle(a)cines.fr
|
|
|
=3D20
------=3D_Part_378524_1670661841.1465374276865--
--===============0514765152618994760==
Content-Type: multipart/alternative
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.bin"
LS0tLS0tPV9QYXJ0XzM3ODUyNF8xNjcwNjYxODQxLjE0NjUzNzQyNzY4NjUKQ29udGVudC1UeXBl
OiB0ZXh0L3BsYWluOyBjaGFyc2V0PXV0Zi04CkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1
b3RlZC1wcmludGFibGUKClRoYW5rcyB0byBhbGwgb2YgdXMgIT0yMApXZSByZWFkIHRoZSBTYW5k
cm8ncyBlbWFpbCBhbm5vdWNlbWVudDo9MjAKaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9waXBlcm1h
aWwvdXNlcnMvMjAxNi1KdW5lLzA0MDE3Mi5odG1sPTIwCkFuZCBhcHBsaWVkIHRvIHRoZSBIViB0
aGUgdXBkYXRlIHByb2Nlc3MgcG9pbnRlZCBpbjo9MjAKaHR0cDovL3d3dy5vdmlydC5vcmcvcmVs
ZWFzZS8zLjYuNy89MjAKKHdlIG9ubHkgdXBkYXRlZCB2ZHNtKT0yMApOb3cgb3VyIEhWIGlzIHdv
cmtpbmcgZmluZSBhbmQgd2UgYXJlIGFibGUgYWdhaW4gdG8gZG8gcGNpIHBhc3N0aHJvdWdoIHdp
dGg9CiBpdC49MjAKQXMgYSByZXN1bHQsIHRoZSB2aXJ0dWFsIG1hY2hpbmUgaG9zdGVkIG9uIGl0
IG5vdyBzZWVzIHRhcGUuPTIwClRoYW5rcyBhZ2FpbiBmb3IgeW91ciBmYXN0IGFuZCBwcmVjaXNv
dXMgaGVscCAhPTIwCkplcm9tZT0yMAoKCkZyb206ICJNaWNoYWwgU2tyaXZhbmVrIiA8bWljaGFs
LnNrcml2YW5la0ByZWRoYXQuY29tPj0yMApUbzogIkxpb25lbCBDYWlnbmVjIiA8Y2FpZ25lY0Bj
aW5lcy5mcj49MjAKQ2M6ICJob252YXVsdCIgPGhvbnZhdWx0QGNpbmVzLmZyPiwgIk1hcnRpbiBQ
b2xlZG5payIgPG1wb2xlZG5pa0ByZWRoYXQuY29tPQo+LCAidXNlcnMiIDx1c2Vyc0BvdmlydC5v
cmc+PTIwClNlbnQ6IFdlZG5lc2RheSwgSnVuZSA4LCAyMDE2IDk6MTI6NDQgQU09MjAKU3ViamVj
dDogUmU6IFtvdmlydC11c2Vyc10gUENJIFBhc3N0aHJvdWdoIHByb2JsZW09MjAKCj4gT24gMDgg
SnVuIDIwMTYsIGF0IDA4OjQxLCBMaW9uZWwgQ2FpZ25lYyA8Y2FpZ25lY0BjaW5lcy5mcj4gd3Jv
dGU6PTIwCj49MjAKPiBIaSB0aGFua3MgZm9yIHJlcGx5PTIwCj49MjAKPiBUaGVyZSBpcyBzb21l
dGhpbmcgaSBkb2VzIG5vdCB1bmRlcnN0YW5kLCBvbiBteSBob3N0cyBpJ3ZlIHZkc20gNC4xNy4y
OC0wPQouZWw3IG5vIGluZGljYXRpb24gYWJvdXQgYSB2ZXJzaW9uIDMuNi41IG9yIDMuNi43Lj0y
MAo+IEhvdyBjYW4gaSBnZXQgdGhpcyB2ZXJzaW9uPz0yMAoKVGhlIG92aXJ0IHJlbGVhc2UgMy42
LjcgcmMyLCBzZWUgU2FuZHJvPUUyPTgwPTk5cyBhbm5vdW5jZW1lbnQgZW1haWwgc2VudCB0PQpv
IHRoaXMgbGlzdCBsYXN0IFRodXJzZGF5LiBPbmNlIHlvdSByZWNvbmZpZ3VyZSByZXBvcyBhcyBl
eHBsYWluZWQgdGhlcmUgeW89CnUgd2lsbCBzZWUgdGhlIG5ldyBmaXhlZCB2ZHNtOikgSXQ9RTI9
ODA9OTlzIGEgcmVsZWFzZSBjYW5kaWRhdGUgcmVwbz0yMAoKSG9wZSBpdCB3b3Jrcyw9MjAKbWlj
aGFsPTIwCgo+PTIwCj49MjAKPj0yMAo+PTIwCj4gLS0tLS0gTWFpbCBvcmlnaW5hbCAtLS0tLT0y
MAo+IERlOiAiTWFydGluIFBvbGVkbmlrIiA8bXBvbGVkbmlrQHJlZGhhdC5jb20+PTIwCj4gPUMz
PTgwOiAiTGlvbmVsIENhaWduZWMiIDxjYWlnbmVjQGNpbmVzLmZyPj0yMAo+IENjOiAiTWFydGlu
IFBvbGVkbmlrIiA8bXBvbGVkbmlrQHJlZGhhdC5jb20+LCAiY2hhcGVsbGUiIDxjaGFwZWxsZUBj
aW5lcy49CmZyPiwgInVzZXJzIiA8dXNlcnNAb3ZpcnQub3JnPiwgImhvbnZhdWx0IiA8aG9udmF1
bHRAY2luZXMuZnI+PTIwCj4gRW52b3k9QzM9QTk6IE1lcmNyZWRpIDggSnVpbiAyMDE2IDA4OjIw
OjAyPTIwCj4gT2JqZXQ6IFJlOiBQQ0kgUGFzc3Rocm91Z2ggcHJvYmxlbT0yMAo+PTIwCj4gT24g
MDcvMDYvMTYgMTk6MDQgKzAyMDAsIExpb25lbCBDYWlnbmVjIHdyb3RlOj0yMAo+PiBIaSBpJ20g
Y293b3JrZXIgb2YgTXIgQ2hhcGVsbGUsPTIwCj4+PTIwCj4+IFRoYW5rIHlvdSBmb3IgeW91ciBy
ZXBseS49MjAKPj49MjAKPj4gc28gaWYgaSB1bmRlcnN0YW5kIHdlIG5lZWQgdG8gImRvd25ncmFk
ZSIgb3VyIG92aXJ0IGVuZ2luZSB3aGljaCBpcyBpbiB2PQplcnNpb24gMy42LjYueCB0byB2ZXJz
aW9uIDMuNi41Pz0yMAo+PiBCdXQgaG93IGNhbiB3ZSBkbyB0aGF0Pz0yMAo+PTIwCj4gWW91IHdv
dWxkIG9ubHkgbmVlZCB0byBkb3duZ3JhZGUgdGhlIDIgaG9zdHMgdG8gMy42LjUgVkRTTS4gQnV0
PTIwCj4gdGhpbmtpbmcgb2YgaXQsIHlvdSBjb3VsZCBpbnN0ZWFkIHVwZ3JhZGUgdG8gMy42Ljcg
KHJjMikgLSB0aGUgYnVnIGlzPTIwCj4gZml4ZWQgdGhlcmUuPTIwCj49MjAKPj4gTGlvbmVsIENh
aWduZWMuPTIwCj4+PTIwCj4+PTIwCj4+PTIwCj4+IC0tLS0tIE1haWwgb3JpZ2luYWwgLS0tLS09
MjAKPj4gRGU6ICJNYXJ0aW4gUG9sZWRuaWsiIDxtcG9sZWRuaWtAcmVkaGF0LmNvbT49MjAKPj4g
PUMzPTgwOiAiY2hhcGVsbGUiIDxjaGFwZWxsZUBjaW5lcy5mcj49MjAKPj4gQ2M6ICJ1c2VycyIg
PHVzZXJzQG92aXJ0Lm9yZz4sICJob252YXVsdCIgPGhvbnZhdWx0QGNpbmVzLmZyPj0yMAo+PiBF
bnZveT1DMz1BOTogTWFyZGkgNyBKdWluIDIwMTYgMTc6MTE6MTA9MjAKPj4gT2JqZXQ6IFJlOiBb
b3ZpcnQtdXNlcnNdIFBDSSBQYXNzdGhyb3VnaCBwcm9ibGVtPTIwCj4+PTIwCj4+IE9uIDA3LzA2
LzE2IDE2OjUwICswMjAwLCBKPUMzPUE5cj1DMz1CNG1lIENoYXBlbGxlIHdyb3RlOj0yMAo+Pj4g
V2UgYXJlIG5vIG1vcmUgYWJsZSB0byBnaXZlIGEgcGNpIGRldmljZSB0byBvbmUgVk0uPTIwCj4+
PiBIZXJlIGlzIHRoZSBwcm9jZXNzIHdlIHVzZWQgdG8gYXNzaWduIG9uZSBkZXZpY2UgdG8gb25l
IFZNOj0yMAo+Pj4gLSBDbGljayBvbiB0aGUgVk0sIGFuZCB0aGVuIG9uIHRoZSB0YWIgIkhvc3Qg
ZGV2aWNlcyI9MjAKPj4+IC0gSW4gdGhlIHRhYiwgY2xpY2sgb24gInBpbiB0byBob3N0IiBhbmQg
c2VsZWN0IHRoZSBob3N0IHRoYXQgd2lsbCBnaXZlPQogdGhlIHBjaSBkZXZpY2U9MjAKPj4+IC0g
Q2xpY2sgb24gImFkZCBkZXZpY2UiLCBpbiB0aGUgd2luZG93IGFwcGVhciBhbGwgdGhlIHBjaSBk
ZXZpY2VzOiBjaG9vPQpzZSB0aGUgcGNpIGNhcmQgeW91IHdpc2ggKGV4LiBwY2lfMDAwMF84M18w
MF8wIHBjaV8wMDAwXzgzXzAwXzEpIGFuZCB0aGVuIGM9CmxpY2sgb24gdGhlIGdyYXkgYXJyb3cg
dG8gYXNzaWduIHRoaXMgZGV2aWNlIHRvIHRoZSBWTT0yMAo+Pj4gLSBDbGljayBvbiBvaz0yMAo+
Pj4gLSBib290IHRoZSBWTS49MjAKPj4+IFRoZSBwcm9ibGVtIGlzIHRoYXQgaW4gdGhlIHdpbmRv
dyB0aGF0IGFwcGVhcnMsIHRoZXJlIGlzIG5vdCBhIHNpbmdsZSBwPQpjaSBkZXZpY2UgbGlzdGVk
ICh0aGVyZSBzaG91bGQgYmUgbWFueSBsaXN0ZWQpLj0yMAo+Pj4gSSBjaGVja2VkIGZpcnN0OiB0
aGUgSU9NTVUgc2V0dGluZyBpcyBzZXQgaW4gdGhlIGtlcm5lbC49MjAKPj4+PTIwCj4+PiBPdXIg
ZW52aXJvbm1lbnQ6PTIwCj4+PiBXZSBoYXZlIHR3byBIViBydW5uaW5nIENlbnRPczcuPTIwCj4+
PiBFYWNoIG9mIHRoZW0gaGF2ZSB0aHJlZSBmaWJlciBjaGFubmVsIGNhcmRzLiBGaXJzdCBjYXJk
IGlzIGNvbm5lY3RlZCB0bz0KIGEgZGlzayBhcnJheSwgc2Vjb25kIG9uZSB0byB0aGUgZmlyc3Qg
dGFwZSBsaWJyYXJ5IGFuZCB0aGlyZCBvbmUgdG8gYW5vdGhlPQpyIHRhcGUgbGlicmFyeS49MjAK
Pj4+IFdlIHVzZWQgdG8gZ2l2ZSB0aGUgc2Vjb25kIGNhcmQgdG8gYSBWTTEsIGFuZCB0aGUgdGhp
cmQgY2FyZCB0byBhbm90aGVyPQogVk0yLiBCb3RoIFZNcyByYW4gb24gb25lIEhWIG9yIG9uIHRo
ZSBvdGhlciBpbiB0aGUgcGFzdDogZXZlcnl0aGluZyB3YXMgZmk9Cm5lLj0yMAo+Pj4gV2hhdCBo
YXBwZW5lZDo9MjAKPj4+IFRvZGF5IHdlIGFkZCB0byB1cGdyYWRlIG9uZSBvZiB0aGUgSFYgKEhW
MSkuIEJvdGggVk1zIHJhbiBvbiB0aGlzIEhWMS4gPQpXZSB1cGdyYWRlZCB0aGVuIHJlc3RhcnRl
ZCB0aGUgSFYxLj0yMAo+Pj4gVGhlbiBJIGVkaXRlZCB0aGUgc2Vjb25kIFZNMiBpbiBvcmRlciB0
byByZW1vdmUgY2FyZCBmcm9tIEhWMS4gVGhlbiBJIHQ9CnJpZWQgdG8gYWRkIHRoZSBjYXJkIGZy
b20gSFYyIHRvIHRoaXMgVk0yOiB0aGUgbGlzdCBvZiAiaG9zdCBkZXZpY2VzIiBpcyBlbT0KcHR5
Lj0yMAo+Pj4gSSBzdGFydGVkIHRoZSBWTTE6IGl0IHdvcmtzIGZpbmUgKEkgZGlkbid0IGNoYW5n
ZSBhbnl0aGluZyBvbiBpdCkuPTIwCj4+PTIwCj4+IEhlbGxvIT0yMAo+Pj0yMAo+PiBJJ20gYWZy
YWlkIHlvdSBoYXZlIGhpdCB0aGUgYnVnIFsxXS4gRm9yIGhvc3RzIHdoZXJlIHlvdSBuZWVkIGhv
c3Q9MjAKPj4gZGV2aWNlcywgdGhlIGJlc3Qgc29sdXRpb24gaXMgbW9zdCBsaWtlbHkgbm90IHVw
Z3JhZGluZyB0aGVtIHRvIDMuNi42PTIwCj4+IGZvciBub3cuPTIwCj4+PTIwCj4+IFsxXSBodHRw
czovL2J1Z3ppbGxhLnJlZGhhdC5jb20vc2hvd19idWcuY2dpP2lkPTNEMTM0MTI5OT0yMAo+Pj0y
MAo+Pj49MjAKPj4+IC0tPTIwCj4+Pj0yMAo+Pj49MjAKPj4+PTIwCj4+Pj0yMAo+Pj49MjAKPj4+
IEo9QzM9QTlyPUMzPUI0bWUgQ2hhcGVsbGU9MjAKPj4+IEMgZW50cmUgSSBuZm9ybWF0aXF1ZSBO
IGF0aW9uYWwgZGUgbCcgRSBuc2VpZ25lbWVudCBTIHVwPUMzPUE5cmlldXI9MjAKPj4+IDk1MCBy
dWUgZGUgU2FpbnQgUHJpZXN0PTIwCj4+PiAzNCAwOTcgTSBPTlRQRUxMSUVSIENlZGV4IDU9MjAK
Pj4+PTIwCj4+PiBUPUMzPUE5bC4gMDQgNjcgMTQgMTQgMTQ9MjAKPj4+IGVNYWlsIGNoYXBlbGxl
QGNpbmVzLmZyPTIwCj4+Pj0yMAo+Pj0yMAo+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX189MjAKPj4+IFVzZXJzIG1haWxpbmcgbGlzdD0yMAo+Pj4gVXNl
cnNAb3ZpcnQub3JnPTIwCj4+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGlu
Zm8vdXNlcnM9MjAKPj49MjAKPj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX189MjAKPj4gVXNlcnMgbWFpbGluZyBsaXN0PTIwCj4+IFVzZXJzQG92aXJ0Lm9y
Zz0yMAo+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM9MjAK
PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXz0yMAo+IFVz
ZXJzIG1haWxpbmcgbGlzdD0yMAo+IFVzZXJzQG92aXJ0Lm9yZz0yMAo+IGh0dHA6Ly9saXN0cy5v
dmlydC5vcmcvbWFpbG1hbi9saXN0aW5mby91c2Vycz0yMAo+PTIwCj49MjAKCl9fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPTIwClVzZXJzIG1haWxpbmcgbGlz
dD0yMApVc2Vyc0BvdmlydC5vcmc9MjAKaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xp
c3RpbmZvL3VzZXJzPTIwCgotLT0yMAoKCj0wOQo9MDkKPTA5Cj0wOT0wOUo9QzM9QTlyPUMzPUI0
bWUgQ2hhcGVsbGU9MjAKPTA5PTA5QyBlbnRyZSBJIG5mb3JtYXRpcXVlIE4gYXRpb25hbCBkZSBs
JyBFIG5zZWlnbmVtZW50IFMgdXA9QzM9QTlyaWV1cj0KPTIwCjk1MCBydWUgZGUgU2FpbnQgUHJp
ZXN0PTIwCjM0IDA5NyBNIE9OVFBFTExJRVIgQ2VkZXggNT0yMAoKPTA5PTA5VD1DMz1BOWwuID0w
OTA0IDY3IDE0IDE0IDE0PTIwCj0wOT0wOWVNYWlsID0wOWNoYXBlbGxlQGNpbmVzLmZyPTIwCgoK
LS0tLS0tPV9QYXJ0XzM3ODUyNF8xNjcwNjYxODQxLjE0NjUzNzQyNzY4NjUKQ29udGVudC1UeXBl
OiB0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogcXVv
dGVkLXByaW50YWJsZQoKPGh0bWw+PGJvZHk+PGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTogVGlt
ZXMgTmV3IFJvbWFuOyBmb250LXNpemU6IDEycHQ7IGNvPQpsb3I6ICMwMDAwMDAiPjxkaXY+VGhh
bmtzIHRvIGFsbCBvZiB1cyAhPGJyPjwvZGl2PjxkaXY+V2UgcmVhZCB0aGUgU2FuZHJvJ3M9CiBl
bWFpbCBhbm5vdWNlbWVudDo8YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGRpdj5odHRw
Oi8vbGlzdHMub3ZpcnQubz0KcmcvcGlwZXJtYWlsL3VzZXJzLzIwMTYtSnVuZS8wNDAxNzIuaHRt
bDwvZGl2PjxkaXY+QW5kIGFwcGxpZWQgdG8gdGhlIEhWIHRoPQplIHVwZGF0ZSBwcm9jZXNzIHBv
aW50ZWQgaW46PGJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPjwvZGl2PjxkaXY+aHR0cDovL3d3dy49
Cm92aXJ0Lm9yZy9yZWxlYXNlLzMuNi43LzwvZGl2PjxkaXY+KHdlIG9ubHkgdXBkYXRlZCB2ZHNt
KTxiciBkYXRhLW1jZS1ib2d1cz0KPTNEIjEiPjwvZGl2PjxkaXY+Tm93IG91ciBIViBpcyB3b3Jr
aW5nIGZpbmUgYW5kIHdlIGFyZSBhYmxlIGFnYWluIHRvIGRvIHBjPQppIHBhc3N0aHJvdWdoIHdp
dGggaXQuPGJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPjwvZGl2PjxkaXY+QXMgYSByZXN1bHQsIHRo
ZSA9CnZpcnR1YWwgbWFjaGluZSBob3N0ZWQgb24gaXQgbm93IHNlZXMgdGFwZS48YnIgZGF0YS1t
Y2UtYm9ndXM9M0QiMSI+PC9kaXY+PD0KZGl2PlRoYW5rcyBhZ2FpbiBmb3IgeW91ciBmYXN0IGFu
ZCBwcmVjaXNvdXMgaGVscCAhPGJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPQo+PC9kaXY+PGRpdj5K
ZXJvbWU8YnIgZGF0YS1tY2UtYm9ndXM9M0QiMSI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGhyIGlk
PTNEIno9CndjaHIiIGRhdGEtbWFya2VyPTNEIl9fRElWSURFUl9fIj48ZGl2IGRhdGEtbWFya2Vy
PTNEIl9fSEVBREVSU19fIj48Yj5Gcm9tOj0KIDwvYj4iTWljaGFsIFNrcml2YW5layIgJmx0O21p
Y2hhbC5za3JpdmFuZWtAcmVkaGF0LmNvbSZndDs8YnI+PGI+VG86IDwvYj4iPQpMaW9uZWwgQ2Fp
Z25lYyIgJmx0O2NhaWduZWNAY2luZXMuZnImZ3Q7PGJyPjxiPkNjOiA8L2I+ImhvbnZhdWx0IiAm
bHQ7aG9udmE9CnVsdEBjaW5lcy5mciZndDssICJNYXJ0aW4gUG9sZWRuaWsiICZsdDttcG9sZWRu
aWtAcmVkaGF0LmNvbSZndDssICJ1c2VycyIgJj0KbHQ7dXNlcnNAb3ZpcnQub3JnJmd0Ozxicj48
Yj5TZW50OiA8L2I+V2VkbmVzZGF5LCBKdW5lIDgsIDIwMTYgOToxMjo0NCBBTTxiPQpyPjxiPlN1
YmplY3Q6IDwvYj5SZTogW292aXJ0LXVzZXJzXSBQQ0kgUGFzc3Rocm91Z2ggcHJvYmxlbTxicj48
L2Rpdj48YnI+PGQ9Cml2IGRhdGEtbWFya2VyPTNEIl9fUVVPVEVEX1RFWFRfXyI+Jmd0OyBPbiAw
OCBKdW4gMjAxNiwgYXQgMDg6NDEsIExpb25lbCBDYT0KaWduZWMgJmx0O2NhaWduZWNAY2luZXMu
ZnImZ3Q7IHdyb3RlOjxicj4mZ3Q7IDxicj4mZ3Q7IEhpIHRoYW5rcyBmb3IgcmVwbHk8PQpicj4m
Z3Q7IDxicj4mZ3Q7IFRoZXJlIGlzIHNvbWV0aGluZyBpIGRvZXMgbm90IHVuZGVyc3RhbmQsIG9u
IG15IGhvc3RzIGkndmU9CiB2ZHNtIDQuMTcuMjgtMC5lbDcgbm8gaW5kaWNhdGlvbiBhYm91dCBh
IHZlcnNpb24gMy42LjUgb3IgMy42LjcuPGJyPiZndDsgSD0Kb3cgY2FuIGkgZ2V0IHRoaXMgdmVy
c2lvbj88YnI+PGJyPlRoZSBvdmlydCByZWxlYXNlIDMuNi43IHJjMiwgc2VlIFNhbmRybz0KPUUy
PTgwPTk5cyBhbm5vdW5jZW1lbnQgZW1haWwgc2VudCB0byB0aGlzIGxpc3QgbGFzdCBUaHVyc2Rh
eS4gT25jZSB5b3UgcmVjPQpvbmZpZ3VyZSByZXBvcyBhcyBleHBsYWluZWQgdGhlcmUgeW91IHdp
bGwgc2VlIHRoZSBuZXcgZml4ZWQgdmRzbTopIEl0PUUyPQo9ODA9OTlzIGEgcmVsZWFzZSBjYW5k
aWRhdGUgcmVwbzxicj48YnI+SG9wZSBpdCB3b3Jrcyw8YnI+bWljaGFsPGJyPjxicj4mZ3Q9Cjsg
PGJyPiZndDsgPGJyPiZndDsgPGJyPiZndDsgPGJyPiZndDsgLS0tLS0gTWFpbCBvcmlnaW5hbCAt
LS0tLTxicj4mZ3Q7IERlOj0KICJNYXJ0aW4gUG9sZWRuaWsiICZsdDttcG9sZWRuaWtAcmVkaGF0
LmNvbSZndDs8YnI+Jmd0OyA9QzM9ODA6ICJMaW9uZWwgQ2FpPQpnbmVjIiAmbHQ7Y2FpZ25lY0Bj
aW5lcy5mciZndDs8YnI+Jmd0OyBDYzogIk1hcnRpbiBQb2xlZG5payIgJmx0O21wb2xlZG5pa0A9
CnJlZGhhdC5jb20mZ3Q7LCAiY2hhcGVsbGUiICZsdDtjaGFwZWxsZUBjaW5lcy5mciZndDssICJ1
c2VycyIgJmx0O3VzZXJzQG92aT0KcnQub3JnJmd0OywgImhvbnZhdWx0IiAmbHQ7aG9udmF1bHRA
Y2luZXMuZnImZ3Q7PGJyPiZndDsgRW52b3k9QzM9QTk6IE1lcmNyPQplZGkgOCBKdWluIDIwMTYg
MDg6MjA6MDI8YnI+Jmd0OyBPYmpldDogUmU6IFBDSSBQYXNzdGhyb3VnaCBwcm9ibGVtPGJyPiZn
dDs9CiA8YnI+Jmd0OyBPbiAwNy8wNi8xNiAxOTowNCArMDIwMCwgTGlvbmVsIENhaWduZWMgd3Jv
dGU6PGJyPiZndDsmZ3Q7IEhpIGknbT0KIGNvd29ya2VyIG9mIE1yIENoYXBlbGxlLDxicj4mZ3Q7
Jmd0OyA8YnI+Jmd0OyZndDsgVGhhbmsgeW91IGZvciB5b3VyIHJlcGx5PQouPGJyPiZndDsmZ3Q7
IDxicj4mZ3Q7Jmd0OyBzbyBpZiBpIHVuZGVyc3RhbmQgd2UgbmVlZCB0byAiZG93bmdyYWRlIiBv
dXIgb3Y9CmlydCBlbmdpbmUgd2hpY2ggaXMgaW4gdmVyc2lvbiAzLjYuNi54IHRvIHZlcnNpb24g
My42LjU/PGJyPiZndDsmZ3Q7IEJ1dCBobz0KdyBjYW4gd2UgZG8gdGhhdD88YnI+Jmd0OyA8YnI+
Jmd0OyBZb3Ugd291bGQgb25seSBuZWVkIHRvIGRvd25ncmFkZSB0aGUgMiBoPQpvc3RzIHRvIDMu
Ni41IFZEU00uIEJ1dDxicj4mZ3Q7IHRoaW5raW5nIG9mIGl0LCB5b3UgY291bGQgaW5zdGVhZCB1
cGdyYWRlIHQ9Cm8gMy42LjcgKHJjMikgLSB0aGUgYnVnIGlzPGJyPiZndDsgZml4ZWQgdGhlcmUu
PGJyPiZndDsgPGJyPiZndDsmZ3Q7IExpb25lbD0KIENhaWduZWMuPGJyPiZndDsmZ3Q7IDxicj4m
Z3Q7Jmd0OyA8YnI+Jmd0OyZndDsgPGJyPiZndDsmZ3Q7IC0tLS0tIE1haWwgb3JpPQpnaW5hbCAt
LS0tLTxicj4mZ3Q7Jmd0OyBEZTogIk1hcnRpbiBQb2xlZG5payIgJmx0O21wb2xlZG5pa0ByZWRo
YXQuY29tJmd0Ozw9CmJyPiZndDsmZ3Q7ID1DMz04MDogImNoYXBlbGxlIiAmbHQ7Y2hhcGVsbGVA
Y2luZXMuZnImZ3Q7PGJyPiZndDsmZ3Q7IENjOiAidT0Kc2VycyIgJmx0O3VzZXJzQG92aXJ0Lm9y
ZyZndDssICJob252YXVsdCIgJmx0O2hvbnZhdWx0QGNpbmVzLmZyJmd0Ozxicj4mZ3Q7PQomZ3Q7
IEVudm95PUMzPUE5OiBNYXJkaSA3IEp1aW4gMjAxNiAxNzoxMToxMDxicj4mZ3Q7Jmd0OyBPYmpl
dDogUmU6IFtvdmlydC09CnVzZXJzXSBQQ0kgUGFzc3Rocm91Z2ggcHJvYmxlbTxicj4mZ3Q7Jmd0
OyA8YnI+Jmd0OyZndDsgT24gMDcvMDYvMTYgMTY6NTAgKz0KMDIwMCwgSj1DMz1BOXI9QzM9QjRt
ZSBDaGFwZWxsZSB3cm90ZTo8YnI+Jmd0OyZndDsmZ3Q7IFdlIGFyZSBubyBtb3JlIGFibGUgPQp0
byBnaXZlIGEgcGNpIGRldmljZSB0byBvbmUgVk0uPGJyPiZndDsmZ3Q7Jmd0OyBIZXJlIGlzIHRo
ZSBwcm9jZXNzIHdlIHVzZWQ9CiB0byBhc3NpZ24gb25lIGRldmljZSB0byBvbmUgVk06PGJyPiZn
dDsmZ3Q7Jmd0OyAtIENsaWNrIG9uIHRoZSBWTSwgYW5kIHRoZT0KbiBvbiB0aGUgdGFiICJIb3N0
IGRldmljZXMiPGJyPiZndDsmZ3Q7Jmd0OyAtIEluIHRoZSB0YWIsIGNsaWNrIG9uICJwaW4gdG8g
PQpob3N0IiBhbmQgc2VsZWN0IHRoZSBob3N0IHRoYXQgd2lsbCBnaXZlIHRoZSBwY2kgZGV2aWNl
PGJyPiZndDsmZ3Q7Jmd0OyAtIEM9CmxpY2sgb24gImFkZCBkZXZpY2UiLCBpbiB0aGUgd2luZG93
IGFwcGVhciBhbGwgdGhlIHBjaSBkZXZpY2VzOiBjaG9vc2UgdGhlID0KcGNpIGNhcmQgeW91IHdp
c2ggKGV4LiBwY2lfMDAwMF84M18wMF8wIHBjaV8wMDAwXzgzXzAwXzEpIGFuZCB0aGVuIGNsaWNr
IG9uPQogdGhlIGdyYXkgYXJyb3cgdG8gYXNzaWduIHRoaXMgZGV2aWNlIHRvIHRoZSBWTTxicj4m
Z3Q7Jmd0OyZndDsgLSBDbGljayBvbiA9Cm9rPGJyPiZndDsmZ3Q7Jmd0OyAtIGJvb3QgdGhlIFZN
Ljxicj4mZ3Q7Jmd0OyZndDsgVGhlIHByb2JsZW0gaXMgdGhhdCBpbiB0aD0KZSB3aW5kb3cgdGhh
dCBhcHBlYXJzLCB0aGVyZSBpcyBub3QgYSBzaW5nbGUgcGNpIGRldmljZSBsaXN0ZWQgKHRoZXJl
IHNob3VsPQpkIGJlIG1hbnkgbGlzdGVkKS48YnI+Jmd0OyZndDsmZ3Q7IEkgY2hlY2tlZCBmaXJz
dDogdGhlIElPTU1VIHNldHRpbmcgaXMgc2U9CnQgaW4gdGhlIGtlcm5lbC48YnI+Jmd0OyZndDsm
Z3Q7IDxicj4mZ3Q7Jmd0OyZndDsgT3VyIGVudmlyb25tZW50Ojxicj4mZ3Q7Jj0KZ3Q7Jmd0OyBX
ZSBoYXZlIHR3byBIViBydW5uaW5nIENlbnRPczcuPGJyPiZndDsmZ3Q7Jmd0OyBFYWNoIG9mIHRo
ZW0gaGF2ZSB0PQpocmVlIGZpYmVyIGNoYW5uZWwgY2FyZHMuIEZpcnN0IGNhcmQgaXMgY29ubmVj
dGVkIHRvIGEgZGlzayBhcnJheSwgc2Vjb25kIG89Cm5lIHRvIHRoZSBmaXJzdCB0YXBlIGxpYnJh
cnkgYW5kIHRoaXJkIG9uZSB0byBhbm90aGVyIHRhcGUgbGlicmFyeS48YnI+Jmd0Oz0KJmd0OyZn
dDsgV2UgdXNlZCB0byBnaXZlIHRoZSBzZWNvbmQgY2FyZCB0byBhIFZNMSwgYW5kIHRoZSB0aGly
ZCBjYXJkIHRvIGFuPQpvdGhlciBWTTIuIEJvdGggVk1zIHJhbiBvbiBvbmUgSFYgb3Igb24gdGhl
IG90aGVyIGluIHRoZSBwYXN0OiBldmVyeXRoaW5nIHc9CmFzIGZpbmUuPGJyPiZndDsmZ3Q7Jmd0
OyBXaGF0IGhhcHBlbmVkOjxicj4mZ3Q7Jmd0OyZndDsgVG9kYXkgd2UgYWRkIHRvIHVwZz0KcmFk
ZSBvbmUgb2YgdGhlIEhWIChIVjEpLiBCb3RoIFZNcyByYW4gb24gdGhpcyBIVjEuIFdlIHVwZ3Jh
ZGVkIHRoZW4gcmVzdGFyPQp0ZWQgdGhlIEhWMS48YnI+Jmd0OyZndDsmZ3Q7IFRoZW4gSSBlZGl0
ZWQgdGhlIHNlY29uZCBWTTIgaW4gb3JkZXIgdG8gcmVtb3Y9CmUgY2FyZCBmcm9tIEhWMS4gVGhl
biBJIHRyaWVkIHRvIGFkZCB0aGUgY2FyZCBmcm9tIEhWMiB0byB0aGlzIFZNMjogdGhlIGxpcz0K
dCBvZiAiaG9zdCBkZXZpY2VzIiBpcyBlbXB0eS48YnI+Jmd0OyZndDsmZ3Q7IEkgc3RhcnRlZCB0
aGUgVk0xOiBpdCB3b3JrcyBmPQppbmUgKEkgZGlkbid0IGNoYW5nZSBhbnl0aGluZyBvbiBpdCku
PGJyPiZndDsmZ3Q7IDxicj4mZ3Q7Jmd0OyBIZWxsbyE8YnI+Jmc9CnQ7Jmd0OyA8YnI+Jmd0OyZn
dDsgSSdtIGFmcmFpZCB5b3UgaGF2ZSBoaXQgdGhlIGJ1ZyBbMV0uIEZvciBob3N0cyB3aGVyZSB5
bz0KdSBuZWVkIGhvc3Q8YnI+Jmd0OyZndDsgZGV2aWNlcywgdGhlIGJlc3Qgc29sdXRpb24gaXMg
bW9zdCBsaWtlbHkgbm90IHVwZ3JhPQpkaW5nIHRoZW0gdG8gMy42LjY8YnI+Jmd0OyZndDsgZm9y
IG5vdy48YnI+Jmd0OyZndDsgPGJyPiZndDsmZ3Q7IFsxXSBodHRwczo9Ci8vYnVnemlsbGEucmVk
aGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9M0QxMzQxMjk5PGJyPiZndDsmZ3Q7IDxicj4mZ3Q7Jmd0
OyZndD0KOyA8YnI+Jmd0OyZndDsmZ3Q7IC0tPGJyPiZndDsmZ3Q7Jmd0OyA8YnI+Jmd0OyZndDsm
Z3Q7IDxicj4mZ3Q7Jmd0OyZndDsgJm5iPQpzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDs8YnI+Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyY9Cm5ic3A7Jm5ic3A7
Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7PGJyPiZndDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcD0KOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzxicj4mZ3Q7Jmd0OyZndDsgJm5ic3A7
Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5iPQpzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtKPUMzPUE5cj1DMz0KPUI0bWUgQ2hh
cGVsbGU8YnI+Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw
OyZuYnNwOyZuPQpic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7
Jm5ic3A7QyBlbnRyZSBJIG5mb3JtYXRpcXVlIE49CiBhdGlvbmFsIGRlIGwnIEUgbnNlaWduZW1l
bnQgUyB1cD1DMz1BOXJpZXVyPGJyPiZndDsmZ3Q7Jmd0OyA5NTAgcnVlIGRlIFNhaT0KbnQgUHJp
ZXN0PGJyPiZndDsmZ3Q7Jmd0OyAzNCAwOTcgTSBPTlRQRUxMSUVSIENlZGV4IDU8YnI+Jmd0OyZn
dDsmZ3Q7IDxicj4mPQpndDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmI9CnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwO1Q9QzM9QTlsLiAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbj0KYnNw
OyZuYnNwOyZuYnNwOzA0IDY3IDE0IDE0IDE0PGJyPiZndDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzPQpwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO2VNYWlsICY9Cm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Y2hhcGVsbGVAY2luZXMuZnI8YnI+Jmd0OyZndD0K
OyZndDsgPGJyPiZndDsmZ3Q7IDxicj4mZ3Q7Jmd0OyZndDsgX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fPQpfX19fX19fXzxicj4mZ3Q7Jmd0OyZndDsgVXNlcnMgbWFpbGlu
ZyBsaXN0PGJyPiZndDsmZ3Q7Jmd0OyBVc2Vyc0BvdmlydC5vcmc9Cjxicj4mZ3Q7Jmd0OyZndDsg
aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPGJyPiZndDsmZ3Q7
ID0KPGJyPiZndDsmZ3Q7IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fPGJyPiZndDsmZ3Q7IFVzPQplcnMgbWFpbGluZyBsaXN0PGJyPiZndDsmZ3Q7IFVzZXJz
QG92aXJ0Lm9yZzxicj4mZ3Q7Jmd0OyBodHRwOi8vbGlzdHMub3ZpcnQ9Ci5vcmcvbWFpbG1hbi9s
aXN0aW5mby91c2Vyczxicj4mZ3Q7IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fXz0KX19fX19fX188YnI+Jmd0OyBVc2VycyBtYWlsaW5nIGxpc3Q8YnI+Jmd0OyBVc2Vyc0Bv
dmlydC5vcmc8YnI+Jmd0OyBodHRwOi8vPQpsaXN0cy5vdmlydC5vcmcvbWFpbG1hbi9saXN0aW5m
by91c2Vyczxicj4mZ3Q7IDxicj4mZ3Q7IDxicj48YnI+X19fX19fX19fX189Cl9fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fXzxicj5Vc2VycyBtYWlsaW5nIGxpc3Q8YnI+VXNlcnNA
b3ZpcnQubz0Kcmc8YnI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3Vz
ZXJzPGJyPjwvZGl2Pjxicj48ZGl2IGRhdGEtPQptYXJrZXI9M0QiX19TSUdfUE9TVF9fIj4tLSA8
YnI+PC9kaXY+PGRpdj48YnI+CiAgICAgIDx0YWJsZSBzdHlsZT0zRCJib3JkZXI6IDJweCBzb2xp
ZCI+CiAgICAgICAgPHRib2R5PgogICAgICAgICAgPHRyPgogICAgICAgICAgICA8dGQ+CiAgICAg
ICAgICAgICAgPHRhYmxlIGNlbGxwYWRkaW5nPTNEIjAiIGNlbGxzcGFjaW5nPTNEIjAiIGJvcmRl
cj0zRCIwIj4KICAgICAgICAgICAgICAgIDx0Ym9keT4KICAgICAgICAgICAgICAgICAgPHRyPgog
ICAgICAgICAgICAgICAgICAgIDx0ZD48YSBocmVmPTNEImh0dHA6Ly93d3cuY2luZXMuZnIiPjxp
bWcgc3R5bGU9M0QiYm9yZGU9CnI6IDBweCBub25lOyIgc3JjPTNEImh0dHBzOi8vd3d3LmNpbmVz
LmZyL3dwLWNvbnRlbnQvdXBsb2Fkcy8yMDE0LzAxL2xvZ28tcz0KaXRlMjExLnBuZyIgaGVpZ2h0
PTNEIjQzIiB3aWR0aD0zRCI2MCI+PC9hPjxiciBkYXRhLW1jZS1ib2d1cz0zRCIxIj48L3RkPgog
ICAgICAgICAgICAgICAgICAgIDx0ZD4KICAgICAgICAgICAgICAgICAgICAgIDx0YWJsZSBjZWxs
cGFkZGluZz0zRCIwIiBjZWxsc3BhY2luZz0zRCIwIj4KICAgICAgICAgICAgICAgICAgICAgICAg
PHRib2R5PgogICAgICAgICAgICAgICAgICAgICAgICAgIDx0cj4KICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIDx0ZCBzdHlsZT0zRCJib3JkZXItYm90dG9tOiAxcHggc29saWQKICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgO2JvcmRlci1sZWZ0OiAxcHggc29saWQgOyBwYWRkaW5nLXRv
cDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMnB4OyBwYWRkaW5nLWJvdHRvbTogMnB4
OyIgd2lkdGg9M0QiMzJweCI+Jm5iPQpzcDs8L3RkPgogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgPHRkIHN0eWxlPTNEImJvcmRlci1ib3R0b206IDFweCBzb2xpZCA7IiBjb2xzcGE9Cm49M0Qi
MiI+PGJpZz5KPUMzPUE5cj1DMz1CNG1lIENoYXBlbGxlPC9iaWc+PC90ZD4KICAgICAgICAgICAg
ICAgICAgICAgICAgICA8L3RyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDx0cj4KICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIDx0ZCBzdHlsZT0zRCJib3JkZXItYm90dG9tOiAxcHggc29s
aWQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgO2JvcmRlci1sZWZ0OiAxcHggc29saWQg
OyBwYWRkaW5nLXRvcDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMnB4OyBwYWRkaW5n
LWJvdHRvbTogMnB4OyIgd2lkdGg9M0QiMzJweCI+Jm5iPQpzcDs8L3RkPgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgPHRkIHN0eWxlPTNEImJvcmRlci1ib3R0b206IDFweCBzb2xpZCA7IiBj
b2xzcGE9Cm49M0QiMiI+IDxiPkM8L2I+PHNtYWxsPmVudHJlIDwvc21hbGw+PGI+STwvYj48c21h
bGw+bmZvcm1hdGlxdWUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC9zbWFsbD48Yj5O
PC9iPjxzbWFsbD5hdGlvbmFsIGRlIGwnPC9zbWFsbD48PQpiPkU8L2I+PHNtYWxsPm5zZWlnbmVt
ZW50CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvc21hbGw+PGI+UzwvYj48c21hbGw+
dXA9QzM9QTlyaWV1cjwvc21hbGw+PD0KYnI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IDxzbWFsbD45NTAgcnVlIGRlIFNhaW50IFByaWVzdDwvc21hbGw+PGJyPgogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAzNDxzbWFsbD4wOTc8L3NtYWxsPiBNPHNtYWxsPk9OVFBFTExJRVIK
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBDZWRleCA1PC9zbWFsbD48YnI+CiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICA8L3RkPgogICAgICAgICAgICAgICAgICAgICAgICAgIDwv
dHI+CiAgICAgICAgICAgICAgICAgICAgICAgICAgPHRyPgogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgPHRkIHN0eWxlPTNEImJvcmRlci1sZWZ0OiAxcHggc29saWQgOwogICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICBwYWRkaW5nLXRvcDogMnB4OyAiIHdpZHRoPTNEIjMycHgiPiZuYnNw
OzwvdGQ9Cj4KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDx0ZD5UPUMzPUE5bC48L3RkPgog
ICAgICAgICAgICAgICAgICAgICAgICAgICAgPHRkPjA0IDY3IDE0IDE0IDE0PC90ZD4KICAgICAg
ICAgICAgICAgICAgICAgICAgICA8L3RyPgogICAgICAgICAgICAgICAgICAgICAgICAgIDx0cj4K
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDx0ZCBzdHlsZT0zRCJib3JkZXItYm90dG9tOiAx
cHggc29saWQKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgO2JvcmRlci1sZWZ0OiAxcHgg
c29saWQgOyBwYWRkaW5nLWJvdHRvbToKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMnB4
OyIgd2lkdGg9M0QiMzJweCI+Jm5ic3A7PC90ZD4KICAgICAgICAgICAgICAgICAgICAgICAgICAg
IDx0ZCBzdHlsZT0zRCJib3JkZXItYm90dG9tOiAxcHggc29saWQgOyI+ZU1haWw8PQovdGQ+CiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICA8dGQgc3R5bGU9M0QiYm9yZGVyLWJvdHRvbTogMXB4
IHNvbGlkICI+PGEgaHJlZj0KPTNEIm1haWx0bzpjaGFwZWxsZUBjaW5lcy5mciI+Y2hhcGVsbGVA
Y2luZXMuZnI8L2E+PGJyIGRhdGEtbWNlLWJvZ3VzPTNEIjEiPQo+PC90ZD4KICAgICAgICAgICAg
ICAgICAgICAgICAgICA8L3RyPgogICAgICAgICAgICAgICAgICAgICAgICA8L3Rib2R5PgogICAg
ICAgICAgICAgICAgICAgICAgPC90YWJsZT4KICAgICAgICAgICAgICAgICAgICA8L3RkPgogICAg
ICAgICAgICAgICAgICA8L3RyPgogICAgICAgICAgICAgICAgPC90Ym9keT4KICAgICAgICAgICAg
ICA8L3RhYmxlPgogICAgICAgICAgICA8L3RkPgogICAgICAgICAgPC90cj4KICAgICAgICA8L3Ri
b2R5PgogICAgICA8L3RhYmxlPgogICA9MjAKICAgIDxicj48L2Rpdj48L2Rpdj48L2JvZHk+PC9o
dG1sPgotLS0tLS09X1BhcnRfMzc4NTI0XzE2NzA2NjE4NDEuMTQ2NTM3NDI3Njg2NS0tCg==
--===============0514765152618994760==--