11:01 a.m.
--_000_8C2CD99696B62B409EBEBFDF87C83FBF59B5604A10DENU1XMAIL02p_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Further info about this case:
An already installed and running VM (Centos7) with static IPv4 assignment, =
if changed to DHCP mode, do not acquire the IP address.
In this case, tcpdump taken on the VM, do not show DHCP offer packets that =
are instead seen on the host bond interface.
Seems that something is filtering DHCP offers between host physical eth int=
erfaces and VM virtio eth interface.
Physical servers on the same VLAN keep DHCP offers and boot from PXE correc=
tly.
Roberto
<head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Dus-ascii"=
<meta name=3D"Generator" content=3D"Microsoft
Word 14 (filtered medium)" <style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.StileMessaggioDiPostaElettronica17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.StileMessaggioDiPostaElettronica18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" / </xml><![endif]--><!--[if gte mso
9]><xml <o:shapelayout
v:ext=3D"edit" <o:idmap v:ext=3D"edit"
data=3D"1" /
</o:shapelayout></xml><![endif]--
</head <body lang=3D"IT"
link=3D"blue" vlink=3D"purple" <div
class=3D"WordSection1" <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">Further=
info about this case:<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">An alre=
ady installed and running VM (Centos7) with static IPv4 assignment, if =
; changed to DHCP mode, do not acquire the IP
address.<o:p></o:p></span></p=
<p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">In this=
case, tcpdump taken on the VM, do not show DHCP offer packets that are ins=
tead seen on the host bond interface.<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">Seems t=
hat something is filtering DHCP offers between host physical eth interfaces=
and VM virtio eth interface.<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US" style=3D"color:#1F497D">Physica=
l servers on the same VLAN keep DHCP offers and boot from PXE correctly.<o:=
p></o:p></span></p <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D">Roberto=
<o:p></o:p></span></p <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D"><o:p>&n=
bsp;</o:p></span></p
<div <p
class=3D"MsoNormal"><span lang=3D"EN-US"
style=3D"color:#1F497D;mso-fare=
ast-language:IT"><o:p> </o:p></span></p </div <p
class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Hi all<o:p></o:p></sp=
an></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">We are using oVirt en=
gine 3.5.1-0.0 on Centos 6.6<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">We are deploying two =
hosts with
vdsm-4.16.10-8.gitc937927.el7.x86_64<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">No hosted-engine, it =
run on a dedicates VM, outside oVirt.<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Behavior: PXE boot of=
a VM, ends in timeout (0x4c106035), instead to accept the DHCP offer comin=
g from DHCP server.<o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Tcpdump capture start=
ed on the vdsm host, bond0 interface shows clearly that DHCP offers reach t=
he vdsm interfaces
<span style=3D"color:#1F497D">></span>three times before PXE
client ends=
in timeout.<o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Incoming DHCP offer i=
s correctly tagged when it comes to the bond0 interface and forwarded to th=
e bond0.bridge
<span
style=3D"color:#1F497D">></span>interface.<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">PXE simply ignore it.=
PXE version is gPXE 0.9.7.<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">bond0.bridge interfac=
e is already setup with STP=3Doff and
DELAY=3D0.<o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">If we install a VM us=
ing command line boot parameters, VM install & run fine. The issu=
e is only related to PXE process,
<span style=3D"color:#1F497D">></span>when it is expected to
use the DHC=
P offer.<o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">I can provide tcpdump=
capture, but I’ve not attached to the email because I’m quite =
new of the community and don’t
<span style=3D"color:#1F497D">></span>know if it is
allowed/correct.<o:p=
></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">On another host, unde=
r the same engine, running vdsm-4.16.12-7.gita30da75.el6.x86_64 on Centos6.=
6, this behavior is
<span style=3D"color:#1F497D">></span>not happening,
everything works fi=
ne.<o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Any idea/suggestion/f=
urther investigation ?<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Thanks for attention<=
o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US" st=
yle=3D"color:#1F497D">></span><span
lang=3D"EN-US">Best regards<o:p></o:=
p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt"><span lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal"
style=3D"margin-left:35.4pt;text-autospace:none"><sp=
an style=3D"font-size:10.0pt;font-family:Consolas;mso-fareast-language:IT">=
Roberto Nunin<o:p></o:p></span></p <p
class=3D"MsoNormal"
style=3D"margin-left:35.4pt;text-autospace:none"><sp=
an style=3D"font-size:10.0pt;font-family:Consolas;mso-fareast-language:IT">=
Infrastructure Manager<o:p></o:p></span></p <p class=3D"MsoNormal"
style=3D"margin-left:35.4pt;text-autospace:none"><sp=
an style=3D"font-size:10.0pt;font-family:Consolas;mso-fareast-language:IT">=
Italy<o:p></o:p></span></p <p
class=3D"MsoNormal" style=3D"margin-left:35.4pt"><span
lang=3D"EN-US"><o=
:p> </o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">Here are interfaces configs:<o:=
p></o:p></span></p <p
class=3D"MsoNormal"><b><span
lang=3D"EN-US">eno1:<o:p></o:p></span></b><=
/p <p class=3D"MsoNormal"><span
lang=3D"EN-US">DEVICE=3D"eno1"<o:p><=
/o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">HWADDR=3D"38:63:bb:4a:47:b=
0"<o:p></o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">MASTER=3D"bond0"<o:p>=
</o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">NM_CONTROLLED=3D"no"<=
o:p></o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">ONBOOT=3D"yes"<o:p></=
o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">SLAVE=3D"yes"<o:p></o=
:p></span></p <p
class=3D"MsoNormal"><b>eno2:<o:p></o:p></b></p <p
class=3D"MsoNormal">DEVICE=3D"eno2"<o:p></o:p></p <p
class=3D"MsoNormal">HWADDR=3D"38:63:bb:4a:47:b4"<o:p></o:p></p=
<p class=3D"MsoNormal"><span
lang=3D"EN-US">MASTER=3D"bond0"<o:p>=
</o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">NM_CONTROLLED=3D"no"<=
o:p></o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">ONBOOT=3D"yes"<o:p></=
o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">SLAVE=3D"yes"<o:p></o=
:p></span></p <p
class=3D"MsoNormal"><b><span
lang=3D"EN-US">bond0:<o:p></o:p></span></b>=
</p <p class=3D"MsoNormal"><span
lang=3D"EN-US">BONDING_OPTS=3D"mode=3D4 m=
iimon=3D100"<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">DEVICE=3D"bond0"<o:p>=
</o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">NM_CONTROLLED=3D"no"<=
o:p></o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">ONBOOT=3D"yes"<o:p></=
o:p></span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">TYPE=3D"Bond"<o:p></o=
:p></span></p <p
class=3D"MsoNormal"><b><span
lang=3D"EN-US">bond0.3500:<o:p></o:p></span=
></b></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">DEVICE=3Dbond0.3500<o:p></o:p><=
/span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">VLAN=3Dyes<o:p></o:p></span></p=
<p class=3D"MsoNormal"><span
lang=3D"EN-US">BRIDGE=3DDMZ3_DEV<o:p></o:p></s=
pan></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">ONBOOT=3Dno<o:p></o:p></span></=
p <p class=3D"MsoNormal"><span
lang=3D"EN-US">MTU=3D1500<o:p></o:p></span></p=
<p class=3D"MsoNormal"><span
lang=3D"EN-US">NM_CONTROLLED=3Dno<o:p></o:p></=
span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">HOTPLUG=3Dno<o:p></o:p></span><=
/p <p class=3D"MsoNormal"><b><span
lang=3D"EN-US">DMZ3_DEV:<o:p></o:p></span><=
/b></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">DEVICE=3DDMZ3_DEV<o:p></o:p></s=
pan></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">TYPE=3DBridge<o:p></o:p></span>=
</p <p class=3D"MsoNormal"><span
lang=3D"EN-US">DELAY=3D0<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">STP=3Doff<o:p></o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US">ONBOOT=3Dno<o:p></o:p></span></=
p <p class=3D"MsoNormal"><span
lang=3D"EN-US">MTU=3D1500<o:p></o:p></span></p=
<p class=3D"MsoNormal"><span
lang=3D"EN-US">DEFROUTE=3Dno<o:p></o:p></span>=
</p <p class=3D"MsoNormal"><span
lang=3D"EN-US">NM_CONTROLLED=3Dno<o:p></o:p></=
span></p <p
class=3D"MsoNormal"><span
lang=3D"EN-US">HOTPLUG=3Dno<o:p></o:p></span><=
/p <p class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p class=3D"MsoNormal"><span
lang=3D"EN-US"><o:p> </o:p></span></p <p
class=3D"MsoNormal"><o:p> </o:p></p </div
<br <hr <font
face=3D"Courier New" color=3D"Black" size=3D"2">Questo
messaggio e' i=
ndirizzato esclusivamente al destinatario indicato e potrebbe contenere inf=
ormazioni confidenziali, riservate o proprietarie. Qualora la presente veni=
sse ricevuta per errore, si prega di segnalarlo
immediatamente al mittente, cancellando l'originale e ogni sua copia e dis=
truggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito=
e potrebbe essere fonte di violazione di legge.<br
<br This message is for the designated
recipient only and may contain privilege=
d, proprietary, or otherwise private information. If you have received it i=
n error, please notify the sender immediately, deleting the original and al=
l copies and destroying any hard
copies. Any other use is strictly prohibited and may be unlawful.<br </font
</body </html
--_000_8C2CD99696B62B409EBEBFDF87C83FBF59B5604A10DENU1XMAIL02p_--
Hi all
We are using oVirt engine 3.5.1-0.0 on Centos 6.6
We are deploying two hosts with vdsm-4.16.10-8.gitc937927.el7.x86_64
No hosted-engine, it run on a dedicates VM, outside oVirt.
Behavior: PXE boot of a VM, ends in timeout (0x4c106035), instead to
accep=
t the DHCP offer coming from DHCP server.
Tcpdump capture started on the vdsm host, bond0 interface shows
clearly th=
at DHCP offers reach the vdsm interfaces >three times before PXE
client end=
s in timeout.
Incoming DHCP offer is correctly tagged when it comes to the bond0
interfa=
ce and forwarded to the bond0.bridge >interface.
PXE simply ignore it. PXE version is gPXE 0.9.7.
bond0.bridge interface is already setup with STP=3Doff and DELAY=3D0.
If we install a VM using command line boot parameters, VM install
& run f=
ine. The issue is only related to PXE process, >when it is expected
to use =
the DHCP offer.
I can provide tcpdump capture, but I've not attached to the email
because =
I'm quite new of the community and don't >know if it is
allowed/correct.
On another host, under the same engine, running
vdsm-4.16.12-7.gita30da75.=
el6.x86_64 on Centos6.6, this behavior is >not
happening, everything works =
fine.
Any idea/suggestion/further investigation ?
Thanks for attention
Best regards
Roberto Nunin
Infrastructure Manager
Italy
Here are interfaces configs:
eno1:
DEVICE=3D"eno1"
HWADDR=3D"38:63:bb:4a:47:b0"
MASTER=3D"bond0"
NM_CONTROLLED=3D"no"
ONBOOT=3D"yes"
SLAVE=3D"yes"
eno2:
DEVICE=3D"eno2"
HWADDR=3D"38:63:bb:4a:47:b4"
MASTER=3D"bond0"
NM_CONTROLLED=3D"no"
ONBOOT=3D"yes"
SLAVE=3D"yes"
bond0:
BONDING_OPTS=3D"mode=3D4 miimon=3D100"
DEVICE=3D"bond0"
NM_CONTROLLED=3D"no"
ONBOOT=3D"yes"
TYPE=3D"Bond"
bond0.3500:
DEVICE=3Dbond0.3500
VLAN=3Dyes
BRIDGE=3DDMZ3_DEV
ONBOOT=3Dno
MTU=3D1500
NM_CONTROLLED=3Dno
HOTPLUG=3Dno
DMZ3_DEV:
DEVICE=3DDMZ3_DEV
TYPE=3DBridge
DELAY=3D0
STP=3Doff
ONBOOT=3Dno
MTU=3D1500
DEFROUTE=3Dno
NM_CONTROLLED=3Dno
HOTPLUG=3Dno
________________________________
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e p=
otrebbe contenere informazioni confidenziali, riservate o proprietarie. Qua=
lora la presente venisse ricevuta per errore, si prega di segnalarlo immedi=
atamente al mittente, cancellando l'originale e ogni sua copia e distruggen=
do eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potr=
ebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privilege=
d, proprietary, or otherwise private information. If you have received it i=
n error, please notify the sender immediately, deleting the original and al=
l copies and destroying any hard copies. Any other use is strictly prohibit=
ed and may be unlawful.
--_000_8C2CD99696B62B409EBEBFDF87C83FBF59B5604A10DENU1XMAIL02p_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40"
1:46 p.m.
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--981903160-1973418136-1431024381=:19841
Content-Type: TEXT/PLAIN; charset=ISO-8859-7; format=flowed
Content-Transfer-Encoding: 8BIT
Hi,
Is your DHCP server a VM running on the same host? I've seem some
strange issues where a VM could not obtain a DHCP lease if it was
running on the same physical machine as the client. If this is the case,
I can look up what I had to change, otherwise ignore it.
Regards,
Rik
Further info about this case:
An already installed and running VM (Centos7) with static IPv4 assignment, if changed to
DHCP mode, do not acquire the
IP address.
In this case, tcpdump taken on the VM, do not show DHCP offer packets that are instead
seen on the host bond interface.
Seems that something is filtering DHCP offers between host physical eth interfaces and VM
virtio eth interface.
Physical servers on the same VLAN keep DHCP offers and boot from PXE correctly.
Roberto
>Hi all
>We are using oVirt engine 3.5.1-0.0 on Centos 6.6
>We are deploying two hosts with vdsm-4.16.10-8.gitc937927.el7.x86_64
>No hosted-engine, it run on a dedicates VM, outside oVirt.
>Behavior: PXE boot of a VM, ends in timeout (0x4c106035), instead to accept the DHCP
offer coming from DHCP server.
>Tcpdump capture started on the vdsm host, bond0 interface shows clearly that DHCP
offers reach the vdsm interfaces
>three times before PXE client ends in timeout.
>Incoming DHCP offer is correctly tagged when it comes to the bond0 interface and
forwarded to the bond0.bridge
>interface.
>PXE simply ignore it. PXE version is gPXE 0.9.7.
>bond0.bridge interface is already setup with STP=off and DELAY=0.
>If we install a VM using command line boot parameters, VM install & run fine.
The issue is only related to PXE
process, >when it is expected to use the DHCP offer.
>I can provide tcpdump capture, but I¢ve not attached to the email because I¢m quite
new of the community and don¢t
>know if it is allowed/correct.
>On another host, under the same engine, running vdsm-4.16.12-7.gita30da75.el6.x86_64
on Centos6.6, this behavior is
>not happening, everything works fine.
>Any idea/suggestion/further investigation ?
>Thanks for attention
>Best regards
Roberto Nunin
Infrastructure Manager
Italy
Here are interfaces configs:
eno1:
DEVICE="eno1"
HWADDR="38:63:bb:4a:47:b0"
MASTER="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
SLAVE="yes"
eno2:
DEVICE="eno2"
HWADDR="38:63:bb:4a:47:b4"
MASTER="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
SLAVE="yes"
bond0:
BONDING_OPTS="mode=4 miimon=100"
DEVICE="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Bond"
bond0.3500:
DEVICE=bond0.3500
VLAN=yes
BRIDGE=DMZ3_DEV
ONBOOT=no
MTU=1500
NM_CONTROLLED=no
HOTPLUG=no
DMZ3_DEV:
DEVICE=DMZ3_DEV
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=no
MTU=1500
DEFROUTE=no
NM_CONTROLLED=no
HOTPLUG=no
_______________________________________________________________________________________________________________________
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni
confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore,
si prega di segnalarlo
immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo
eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise private
information. If you have received it in error, please notify the sender immediately,
deleting the original and all
copies and destroying any hard copies. Any other use is strictly prohibited and may be
unlawful.
--981903160-1973418136-1431024381=:19841--
3:33 a.m.
New subject: R: R: PXE boot of a VM on vdsm don't read DHCP offer
Hi Rick
No, DHCP is provided by an external appliance.
Thanks for answering !
Roberto
-----Messaggio originale-----
Da: Rik Theys [mailto:Rik.Theys@esat.kuleuven.be]
Inviato: giovedì 7 maggio 2015 20:46
A: NUNIN Roberto
Cc: users(a)ovirt.org
Oggetto: Re: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer
Hi,
Is your DHCP server a VM running on the same host? I've seem some
strange issues where a VM could not obtain a DHCP lease if it was
running on the same physical machine as the client. If this is the case,
I can look up what I had to change, otherwise ignore it.
Regards,
Rik
Further info about this case:
An already installed and running VM (Centos7) with static IPv4 assignment, if changed to
DHCP mode, do not acquire the
IP address.
In this case, tcpdump taken on the VM, do not show DHCP offer packets that are instead
seen on the host bond interface.
Seems that something is filtering DHCP offers between host physical eth interfaces and VM
virtio eth interface.
Physical servers on the same VLAN keep DHCP offers and boot from PXE correctly.
Roberto
>Hi all
>We are using oVirt engine 3.5.1-0.0 on Centos 6.6
>We are deploying two hosts with vdsm-4.16.10-8.gitc937927.el7.x86_64
>No hosted-engine, it run on a dedicates VM, outside oVirt.
>Behavior: PXE boot of a VM, ends in timeout (0x4c106035), instead to accept the DHCP
offer coming from DHCP server.
>Tcpdump capture started on the vdsm host, bond0 interface shows clearly that DHCP
offers reach the vdsm interfaces
>three times before PXE client ends in timeout.
>Incoming DHCP offer is correctly tagged when it comes to the bond0 interface and
forwarded to the bond0.bridge
>interface.
>PXE simply ignore it. PXE version is gPXE 0.9.7.
>bond0.bridge interface is already setup with STP=off and DELAY=0.
>If we install a VM using command line boot parameters, VM install & run fine.
The issue is only related to PXE
process, >when it is expected to use the DHCP offer.
>I can provide tcpdump capture, but Iʼve not attached to the email because Iʼm quite
new of the community and donʼt
>know if it is allowed/correct.
>On another host, under the same engine, running vdsm-4.16.12-7.gita30da75.el6.x86_64
on Centos6.6, this behavior is
>not happening, everything works fine.
>Any idea/suggestion/further investigation ?
>Thanks for attention
>Best regards
Roberto Nunin
Infrastructure Manager
Italy
Here are interfaces configs:
eno1:
DEVICE="eno1"
HWADDR="38:63:bb:4a:47:b0"
MASTER="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
SLAVE="yes"
eno2:
DEVICE="eno2"
HWADDR="38:63:bb:4a:47:b4"
MASTER="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
SLAVE="yes"
bond0:
BONDING_OPTS="mode=4 miimon=100"
DEVICE="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Bond"
bond0.3500:
DEVICE=bond0.3500
VLAN=yes
BRIDGE=DMZ3_DEV
ONBOOT=no
MTU=1500
NM_CONTROLLED=no
HOTPLUG=no
DMZ3_DEV:
DEVICE=DMZ3_DEV
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=no
MTU=1500
DEFROUTE=no
NM_CONTROLLED=no
HOTPLUG=no
_______________________________________________________________________________________________________________________
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni
confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore,
si prega di segnalarlo
immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo
eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise private
information. If you have received it in error, please notify the sender immediately,
deleting the original and all
copies and destroying any hard copies. Any other use is strictly prohibited and may be
unlawful.
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
5:08 a.m.
On Thu, May 07, 2015 at 06:01:14PM +0200, NUNIN Roberto wrote:
Further info about this case:
An already installed and running VM (Centos7) with static IPv4 assignment, if changed to
DHCP mode, do not acquire the IP address.
In this case, tcpdump taken on the VM, do not show DHCP offer packets that are instead
seen on the host bond interface.
Seems that something is filtering DHCP offers between host physical eth interfaces and VM
virtio eth interface.
Physical servers on the same VLAN keep DHCP offers and boot from PXE correctly.
Roberto
>Hi all
>We are using oVirt engine 3.5.1-0.0 on Centos 6.6
>We are deploying two hosts with vdsm-4.16.10-8.gitc937927.el7.x86_64
>No hosted-engine, it run on a dedicates VM, outside oVirt.
>Behavior: PXE boot of a VM, ends in timeout (0x4c106035), instead to accept the DHCP
offer coming from DHCP server.
>Tcpdump capture started on the vdsm host, bond0 interface shows clearly that DHCP
offers reach the vdsm interfaces >three times before PXE client ends in timeout.
>Incoming DHCP offer is correctly tagged when it comes to the bond0 interface and
forwarded to the bond0.bridge >interface.
>PXE simply ignore it. PXE version is gPXE 0.9.7.
>bond0.bridge interface is already setup with STP=off and DELAY=0.
>If we install a VM using command line boot parameters, VM install & run fine.
The issue is only related to PXE process, >when it is expected to use the DHCP offer.
>I can provide tcpdump capture, but I've not attached to the email because I'm
quite new of the community and don't >know if it is allowed/correct.
>On another host, under the same engine, running vdsm-4.16.12-7.gita30da75.el6.x86_64
on Centos6.6, this behavior is >not happening, everything works fine.
>Any idea/suggestion/further investigation ?
>Thanks for attention
>Best regards
Which kernel does the el7 host run? I think that Ido has seen a case
where `brctl showmacs` was not populated with the VM mac, despite a
packet coming out of it.
Can you tcpdump and check whether the bridge propogated the DHCP offer
to the tap device of the said VM? Does the packet generated by
`ether-wake MAC-of-VM` reach the tap device?
Roberto Nunin
Infrastructure Manager
Italy
Here are interfaces configs:
eno1:
DEVICE="eno1"
HWADDR="38:63:bb:4a:47:b0"
MASTER="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
SLAVE="yes"
eno2:
DEVICE="eno2"
HWADDR="38:63:bb:4a:47:b4"
MASTER="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
SLAVE="yes"
bond0:
BONDING_OPTS="mode=4 miimon=100"
DEVICE="bond0"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Bond"
bond0.3500:
DEVICE=bond0.3500
VLAN=yes
BRIDGE=DMZ3_DEV
ONBOOT=no
MTU=1500
NM_CONTROLLED=no
HOTPLUG=no
DMZ3_DEV:
DEVICE=DMZ3_DEV
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=no
MTU=1500
DEFROUTE=no
NM_CONTROLLED=no
HOTPLUG=no
8:11 a.m.
New subject: R: R: PXE boot of a VM on vdsm don't read DHCP offer
Hi Dan
Thanks for answering
Which kernel does the el7 host run? I think that Ido has seen a case
where `brctl showmacs` was not populated with the VM mac, despite a
packet coming out of it.
Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't available
within vdsm only package.
Can you tcpdump and check whether the bridge propogated the DHCP offer
to the tap device of the said VM? Does the packet generated by
`ether-wake MAC-of-VM` reach the tap device?
Yes: host "see" the broadcast :
0.000000 0.0.0.0 255.255.255.255 DHCP 346 DHCP
Discover - Transaction ID 0x69267b67
It came from the right MAC:
Source: Qumranet_15:81:03 (00:1a:4a:15:81:03)
And it is tagged correctly:
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
This is the offer, on the bond interface:
1.012355 10.155.124.2 10.155.124.246 DHCP 346 DHCP Offer
- Transaction ID 0x69267b67
Layer 2 info:
Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst: Qumranet_15:81:03
(00:1a:4a:15:81:03)
Tagging on the bond:
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
The tag is correctly removed when DHCP offer is forwarded over the bond.3500.
Here's the offer content, seems everything right:
Client IP address: 0.0.0.0 (0.0.0.0)
Your (client) IP address: 10.155.124.246 (10.155.124.246)
Next server IP address: 10.155.124.223 (10.155.124.223)
Relay agent IP address: 10.155.124.2 (10.155.124.2)
Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03)
Client hardware address padding: 00000000000000000000
Server host name: 10.155.124.223
Boot file name: pxelinux.0
Magic cookie: DHCP
Nothing of this offer appear on the VM side.
ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host)
reach the VM eth0 interface:
2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL 116 MagicPacket
for Qumranet_15:81:03 (00:1a:4a:15:81:03)
Really strange behavior.
Roberto
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
9:14 a.m.
New subject: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Fri, May 08, 2015 at 03:11:25PM +0200, NUNIN Roberto wrote:
Hi Dan
Thanks for answering
> Which kernel does the el7 host run? I think that Ido has seen a case
> where `brctl showmacs` was not populated with the VM mac, despite a
> packet coming out of it.
Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't available
within vdsm only package.
Could you try upgrading to a more up-to-date
http://mirror.centos.org/centos-7/7.1.1503/updates/x86_64/Packages/kernel...
?
bridge-utils is a vdsm dependency. It must exist on your host. Please
see if the mac of the vNIC shows up on `brctl showmacs` as it should.
>
> Can you tcpdump and check whether the bridge propogated the DHCP offer
> to the tap device of the said VM? Does the packet generated by
> `ether-wake MAC-of-VM` reach the tap device?
Yes: host "see" the broadcast :
0.000000 0.0.0.0 255.255.255.255 DHCP 346 DHCP
Discover - Transaction ID 0x69267b67
It came from the right MAC:
Source: Qumranet_15:81:03 (00:1a:4a:15:81:03)
And it is tagged correctly:
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
This is the offer, on the bond interface:
1.012355 10.155.124.2 10.155.124.246 DHCP 346 DHCP
Offer - Transaction ID 0x69267b67
Layer 2 info:
Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst: Qumranet_15:81:03
(00:1a:4a:15:81:03)
Tagging on the bond:
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
The tag is correctly removed when DHCP offer is forwarded over the bond.3500.
Here's the offer content, seems everything right:
Client IP address: 0.0.0.0 (0.0.0.0)
Your (client) IP address: 10.155.124.246 (10.155.124.246)
Next server IP address: 10.155.124.223 (10.155.124.223)
Relay agent IP address: 10.155.124.2 (10.155.124.2)
Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03)
Client hardware address padding: 00000000000000000000
Server host name: 10.155.124.223
Boot file name: pxelinux.0
Magic cookie: DHCP
Nothing of this offer appear on the VM side.
But does it show on the host's bridge? on the tap device?
ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host)
reach the VM eth0 interface:
2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL 116
MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03)
Really strange behavior.
Roberto
3:51 a.m.
New subject: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
Hi Dan, guys
Sorry for very late follow-up, but we had a lot of other topics to fix just before to go
back on this one.
We have tried another approach just to check if the kernel of the vdsm iso image used to
install the host could create the problem I've reported to the list.
Now we have reinstalled the same hardware with latest CentOS 7.1, fully updated.
Installed vdsm, then joined the oVirt cluster.
Well, we are observing the same behavior as before.
No DHCP offer is reaching the booting VM, and:
brctl showmacs <bridge_if> show us the booting vm mac-address
tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.
We have also tried to remove ANY firewall rule.
It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if we install
a vm manually and assign a static IP, it works fine.
If we switch to dhcp, the vm don't get the dynamic one.
In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP offer.
Any further suggestion/hint ?
RN
-----Messaggio originale-----
Da: Dan Kenigsberg [mailto:danken@redhat.com]
Inviato: lunedì 18 maggio 2015 16:14
A: NUNIN Roberto
Cc: users(a)ovirt.org; ibarkan(a)redhat.com
Oggetto: Re: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP
offer
On Fri, May 08, 2015 at 03:11:25PM +0200, NUNIN Roberto wrote:
> Hi Dan
> Thanks for answering
>
>
>
> > Which kernel does the el7 host run? I think that Ido has seen a case
> > where `brctl showmacs` was not populated with the VM mac, despite a
> > packet coming out of it.
>
> Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't
available within vdsm only package.
Could you try upgrading to a more up-to-date
http://mirror.centos.org/centos-
7/7.1.1503/updates/x86_64/Packages/kernel-3.10.0-
229.4.2.el7.x86_64.rpm
?
bridge-utils is a vdsm dependency. It must exist on your host. Please
see if the mac of the vNIC shows up on `brctl showmacs` as it should.
> >
> > Can you tcpdump and check whether the bridge propogated the DHCP
offer
> > to the tap device of the said VM? Does the packet generated by
> > `ether-wake MAC-of-VM` reach the tap device?
>
> Yes: host "see" the broadcast :
> 0.000000 0.0.0.0 255.255.255.255 DHCP 346
DHCP
Discover - Transaction ID 0x69267b67
> It came from the right MAC:
> Source: Qumranet_15:81:03 (00:1a:4a:15:81:03)
> And it is tagged correctly:
> 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
>
> This is the offer, on the bond interface:
> 1.012355 10.155.124.2 10.155.124.246 DHCP 346
DHCP
Offer - Transaction ID 0x69267b67
> Layer 2 info:
> Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst:
Qumranet_15:81:03 (00:1a:4a:15:81:03)
> Tagging on the bond:
> 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
>
> The tag is correctly removed when DHCP offer is forwarded over the
bond.3500.
> Here's the offer content, seems everything right:
>
> Client IP address: 0.0.0.0 (0.0.0.0)
> Your (client) IP address: 10.155.124.246 (10.155.124.246)
> Next server IP address: 10.155.124.223 (10.155.124.223)
> Relay agent IP address: 10.155.124.2 (10.155.124.2)
> Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03)
> Client hardware address padding: 00000000000000000000
> Server host name: 10.155.124.223
> Boot file name: pxelinux.0
> Magic cookie: DHCP
>
> Nothing of this offer appear on the VM side.
But does it show on the host's bridge? on the tap device?
>
> ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host)
> reach the VM eth0 interface:
> 2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL 116
MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03)
>
> Really strange behavior.
>
> Roberto
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
5:31 a.m.
New subject: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Fri, Jul 03, 2015 at 10:51:52AM +0200, NUNIN Roberto wrote:
Hi Dan, guys
Sorry for very late follow-up, but we had a lot of other topics to fix just before to go
back on this one.
We have tried another approach just to check if the kernel of the vdsm iso image used to
install the host could create the problem I've reported to the list.
Now we have reinstalled the same hardware with latest CentOS 7.1, fully updated.
Installed vdsm, then joined the oVirt cluster.
Well, we are observing the same behavior as before.
No DHCP offer is reaching the booting VM, and:
brctl showmacs <bridge_if> show us the booting vm mac-address
tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.
The offer should be replicated to the tap device (vnetXXX). I assume
that it does not?
We have also tried to remove ANY firewall rule.
It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if we
install a vm manually and assign a static IP, it works fine.
If we switch to dhcp, the vm don't get the dynamic one.
In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP offer.
Any further suggestion/hint ?
Which bond mode are you using? I recently seen
https://bugzilla.redhat.com/show_bug.cgi?id=1230638#c22
closed as duplicate of
Bug 1094842 - Bonding modes 0, 5 and 6 should be avoided for VM
networks
9:32 a.m.
New subject: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
Thanks for answering. My considerations below.
BR,
ROberto
-----Messaggio originale-----
Da: Dan Kenigsberg [mailto:danken@redhat.com]
Inviato: venerdì 3 luglio 2015 12:31
A: NUNIN Roberto
Cc: users(a)ovirt.org; ibarkan(a)redhat.com
Oggetto: Re: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
DHCP offer
On Fri, Jul 03, 2015 at 10:51:52AM +0200, NUNIN Roberto wrote:
> Hi Dan, guys
>
> Sorry for very late follow-up, but we had a lot of other topics to fix just
before to go back on this one.
>
> We have tried another approach just to check if the kernel of the vdsm iso
image used to install the host could create the problem I've reported to the
list.
>
> Now we have reinstalled the same hardware with latest CentOS 7.1, fully
updated.
> Installed vdsm, then joined the oVirt cluster.
>
> Well, we are observing the same behavior as before.
> No DHCP offer is reaching the booting VM, and:
>
> brctl showmacs <bridge_if> show us the booting vm mac-address
> tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.
The offer should be replicated to the tap device (vnetXXX). I assume
that it does not?
The DHCP offer package coming from DHCP server flows thru the bond0 device (tagged), the
VLAN device (tagged) and the bridge interface (not anymore tagged, correct), so it seems
that the path back is complete, considering that brctl showmacs shows the vm mac and If we
put a static IP into the vm nic, it works correctly.
>
> We have also tried to remove ANY firewall rule.
>
> It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if we
install a vm manually and assign a static IP, it works fine.
> If we switch to dhcp, the vm don't get the dynamic one.
> In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP
offer.
>
> Any further suggestion/hint ?
We have mode 4, but already checked active/backup, mode 2, without changes.
Which bond mode are you using? I recently seen
https://bugzilla.redhat.com/show_bug.cgi?id=1230638#c22
closed as duplicate of
Bug 1094842 - Bonding modes 0, 5 and 6 should be avoided for VM
networks
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
3:01 a.m.
New subject: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Fri, Jul 03, 2015 at 04:32:50PM +0200, NUNIN Roberto wrote:
Thanks for answering. My considerations below.
BR,
ROberto
> -----Messaggio originale-----
> Da: Dan Kenigsberg [mailto:danken@redhat.com]
> Inviato: venerdì 3 luglio 2015 12:31
> A: NUNIN Roberto
> Cc: users(a)ovirt.org; ibarkan(a)redhat.com
> Oggetto: Re: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
> DHCP offer
>
> On Fri, Jul 03, 2015 at 10:51:52AM +0200, NUNIN Roberto wrote:
> > Hi Dan, guys
> >
> > Sorry for very late follow-up, but we had a lot of other topics to fix just
> before to go back on this one.
> >
> > We have tried another approach just to check if the kernel of the vdsm iso
> image used to install the host could create the problem I've reported to the
> list.
> >
> > Now we have reinstalled the same hardware with latest CentOS 7.1, fully
> updated.
> > Installed vdsm, then joined the oVirt cluster.
> >
> > Well, we are observing the same behavior as before.
> > No DHCP offer is reaching the booting VM, and:
> >
> > brctl showmacs <bridge_if> show us the booting vm mac-address
> > tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.
>
> The offer should be replicated to the tap device (vnetXXX). I assume
> that it does not?
The DHCP offer package coming from DHCP server flows thru the bond0
device (tagged), the VLAN device (tagged) and the bridge interface
(not anymore tagged, correct), so it seems that the path back is
complete, considering that brctl showmacs shows the vm mac and If we
put a static IP into the vm nic, it works correctly.
Let's try to trace the offer for one more link. If it does not get to
the tap device vnetXXX - it is clearly a host-side issue. Maybe a bridge
module bug. If you spot the offer on the tap, the it becomes more likely
that the issue is in qemu or elsewere in the virt stack.
I would appreciate your assistence!
>
> >
> > We have also tried to remove ANY firewall rule.
> >
> > It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if
we
> install a vm manually and assign a static IP, it works fine.
> > If we switch to dhcp, the vm don't get the dynamic one.
> > In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP
> offer.
> >
> > Any further suggestion/hint ?
We have mode 4, but already checked active/backup, mode 2, without changes.
Thanks, so it's not that bonding issue.
Regards,
Dan.
3:33 a.m.
New subject: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
Hi Dan
Sorry for question: what do you mean for interface vnetxxxx ?
Currently our path is :
eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.
Which one of these ?
Moreover, reading Fabian statements about bonding limits, today I can try to switch to a
config without bonding.
BR
Roberto
-----Messaggio originale-----
Da: Dan Kenigsberg [mailto:danken@redhat.com]
Inviato: lunedì 6 luglio 2015 10:02
A: NUNIN Roberto
Cc: users(a)ovirt.org; ibarkan(a)redhat.com
Oggetto: Re: R: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
DHCP offer
On Fri, Jul 03, 2015 at 04:32:50PM +0200, NUNIN Roberto wrote:
> Thanks for answering. My considerations below.
>
> BR,
>
> ROberto
> > -----Messaggio originale-----
> > Da: Dan Kenigsberg [mailto:danken@redhat.com]
> > Inviato: venerdì 3 luglio 2015 12:31
> > A: NUNIN Roberto
> > Cc: users(a)ovirt.org; ibarkan(a)redhat.com
> > Oggetto: Re: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
> > DHCP offer
> >
> > On Fri, Jul 03, 2015 at 10:51:52AM +0200, NUNIN Roberto wrote:
> > > Hi Dan, guys
> > >
> > > Sorry for very late follow-up, but we had a lot of other topics to fix
just
> > before to go back on this one.
> > >
> > > We have tried another approach just to check if the kernel of the vdsm
iso
> > image used to install the host could create the problem I've reported to
the
> > list.
> > >
> > > Now we have reinstalled the same hardware with latest CentOS 7.1,
fully
> > updated.
> > > Installed vdsm, then joined the oVirt cluster.
> > >
> > > Well, we are observing the same behavior as before.
> > > No DHCP offer is reaching the booting VM, and:
> > >
> > > brctl showmacs <bridge_if> show us the booting vm mac-address
> > > tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp
server.
> >
> > The offer should be replicated to the tap device (vnetXXX). I assume
> > that it does not?
>
> The DHCP offer package coming from DHCP server flows thru the bond0
> device (tagged), the VLAN device (tagged) and the bridge interface
> (not anymore tagged, correct), so it seems that the path back is
> complete, considering that brctl showmacs shows the vm mac and If we
> put a static IP into the vm nic, it works correctly.
Let's try to trace the offer for one more link. If it does not get to
the tap device vnetXXX - it is clearly a host-side issue. Maybe a bridge
module bug. If you spot the offer on the tap, the it becomes more likely
that the issue is in qemu or elsewere in the virt stack.
I would appreciate your assistence!
>
> >
> > >
> > > We have also tried to remove ANY firewall rule.
> > >
> > > It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue.
Infact, if
we
> > install a vm manually and assign a static IP, it works fine.
> > > If we switch to dhcp, the vm don't get the dynamic one.
> > > In this case, tcpdump on vm shows only the DHCP discovery, not the
DHCP
> > offer.
> > >
> > > Any further suggestion/hint ?
>
> We have mode 4, but already checked active/backup, mode 2, without
changes.
>
Thanks, so it's not that bonding issue.
Regards,
Dan.
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
11:02 a.m.
New subject: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
Hi Dan
Sorry for question: what do you mean for interface vnetxxxx ?
Currently our path is :
eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.
Which one of these ?
Moreover, reading Fabian statements about bonding limits, today I can try to switch to a
config without bonding.
"vm" is a complicated term.
`brctl show` would not show you a "vm" connected to a bridge. When you
WOULD see is a vnet888 tap device. The "other side" of this device is
held by qemu, which implement the VM.
I'm asking if the dhcp offer has reached that tap device.
3:14 a.m.
New subject: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> Hi Dan
>
> Sorry for question: what do you mean for interface vnetxxxx ?
> Currently our path is :
> eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.
>
> Which one of these ?
> Moreover, reading Fabian statements about bonding limits, today I can try
to switch to a config without bonding.
"vm" is a complicated term.
`brctl show` would not show you a "vm" connected to a bridge. When you
WOULD see is a vnet888 tap device. The "other side" of this device is
held by qemu, which implement the VM.
Ok, understood and found it, vnet2
I'm asking if the dhcp offer has reached that tap device.
No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER.
Roberto
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
11:13 a.m.
New subject: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
>
> On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > Hi Dan
> >
> > Sorry for question: what do you mean for interface vnetxxxx ?
> > Currently our path is :
> > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.
> >
> > Which one of these ?
> > Moreover, reading Fabian statements about bonding limits, today I can try
> to switch to a config without bonding.
>
> "vm" is a complicated term.
>
> `brctl show` would not show you a "vm" connected to a bridge. When you
> WOULD see is a vnet888 tap device. The "other side" of this device is
> held by qemu, which implement the VM.
Ok, understood and found it, vnet2
>
> I'm asking if the dhcp offer has reached that tap device.
No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER.
Ok, so it seems that we have a problem in the host bridging.
Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
returns to the bridge, but is not propagated to the tap device.
Can you suggest how to debug this further?
11:18 a.m.
New subject: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
-----Messaggio originale-----
Da: Dan Kenigsberg [mailto:danken@redhat.com]
Inviato: martedì 7 luglio 2015 18:13
A: NUNIN Roberto; mst(a)redhat.com
Cc: users(a)ovirt.org; ibarkan(a)redhat.com
Oggetto: Re: R: R: R: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
DHCP offer
On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> >
> > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > > Hi Dan
> > >
> > > Sorry for question: what do you mean for interface vnetxxxx ?
> > > Currently our path is :
> > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.
> > >
> > > Which one of these ?
> > > Moreover, reading Fabian statements about bonding limits, today I can
try
> > to switch to a config without bonding.
> >
> > "vm" is a complicated term.
> >
> > `brctl show` would not show you a "vm" connected to a bridge. When
you
> > WOULD see is a vnet888 tap device. The "other side" of this device
is
> > held by qemu, which implement the VM.
>
> Ok, understood and found it, vnet2
>
> >
> > I'm asking if the dhcp offer has reached that tap device.
>
> No, the DHCP offer packet do not reach the vnet2 interface, I can see only
DHCP DISCOVER.
Ok, so it seems that we have a problem in the host bridging.
Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
It is 3.10.0-229.7.2.el7.x86_64 on the host with installed Centos7.1, update, then vdsm
It is 3.10.0-123.20.1.el7.x86_64 on the host installed with vdsm iso
Both hosts show the issue
Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
returns to the bridge, but is not propagated to the tap device.
Can you suggest how to debug this further?
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
1:11 a.m.
New subject: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> >
> > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > > Hi Dan
> > >
> > > Sorry for question: what do you mean for interface vnetxxxx ?
> > > Currently our path is :
> > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.
> > >
> > > Which one of these ?
> > > Moreover, reading Fabian statements about bonding limits, today I can try
> > to switch to a config without bonding.
> >
> > "vm" is a complicated term.
> >
> > `brctl show` would not show you a "vm" connected to a bridge. When
you
> > WOULD see is a vnet888 tap device. The "other side" of this device
is
> > held by qemu, which implement the VM.
>
> Ok, understood and found it, vnet2
>
> >
> > I'm asking if the dhcp offer has reached that tap device.
>
> No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP
DISCOVER.
Ok, so it seems that we have a problem in the host bridging.
Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
returns to the bridge, but is not propagated to the tap device.
Can you suggest how to debug this further?
Dump packets including the ethernet headers.
Likely something interfered with them so the eth address is wrong.
Since bonding does this sometimes, this is the most likely culprit.
--
MST
11:13 a.m.
New subject: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
-----Messaggio originale-----
Da: Michael S. Tsirkin [mailto:mst@redhat.com]
Inviato: mercoledì 8 luglio 2015 08:12
A: Dan Kenigsberg
Cc: NUNIN Roberto; users(a)ovirt.org; ibarkan(a)redhat.com
Oggetto: Re: R: R: R: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
DHCP offer
On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > >
> > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > > > Hi Dan
> > > >
> > > > Sorry for question: what do you mean for interface vnetxxxx ?
> > > > Currently our path is :
> > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge -----
vm.
> > > >
> > > > Which one of these ?
> > > > Moreover, reading Fabian statements about bonding limits, today I
can try
> > > to switch to a config without bonding.
> > >
> > > "vm" is a complicated term.
> > >
> > > `brctl show` would not show you a "vm" connected to a bridge.
When
you
> > > WOULD see is a vnet888 tap device. The "other side" of this
device is
> > > held by qemu, which implement the VM.
> >
> > Ok, understood and found it, vnet2
> >
> > >
> > > I'm asking if the dhcp offer has reached that tap device.
> >
> > No, the DHCP offer packet do not reach the vnet2 interface, I can see only
DHCP DISCOVER.
>
> Ok, so it seems that we have a problem in the host bridging.
>
> Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
>
> Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
> returns to the bridge, but is not propagated to the tap device.
> Can you suggest how to debug this further?
Dump packets including the ethernet headers.
I have both tcpdump capture, on the bridge interface and on the vnet interface.
How I can upload output files ?
Sorry for probably obvious question, first time for me :)
Roberto Nunin
Likely something interfered with them so the eth address is wrong.
Since bonding does this sometimes, this is the most likely culprit.
--
MST
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
7:16 a.m.
New subject: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > >
> > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > > > Hi Dan
> > > >
> > > > Sorry for question: what do you mean for interface vnetxxxx ?
> > > > Currently our path is :
> > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge -----
vm.
> > > >
> > > > Which one of these ?
> > > > Moreover, reading Fabian statements about bonding limits, today I can
try
> > > to switch to a config without bonding.
> > >
> > > "vm" is a complicated term.
> > >
> > > `brctl show` would not show you a "vm" connected to a bridge.
When you
> > > WOULD see is a vnet888 tap device. The "other side" of this
device is
> > > held by qemu, which implement the VM.
> >
> > Ok, understood and found it, vnet2
> >
> > >
> > > I'm asking if the dhcp offer has reached that tap device.
> >
> > No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP
DISCOVER.
>
> Ok, so it seems that we have a problem in the host bridging.
>
> Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
>
> Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
> returns to the bridge, but is not propagated to the tap device.
> Can you suggest how to debug this further?
Dump packets including the ethernet headers.
Likely something interfered with them so the eth address is wrong.
Since bonding does this sometimes, this is the most likely culprit.
We've ruled this out already - Roberto reproduces the issue without a
bond.
7:57 a.m.
New subject: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
----- Original Message -----
On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > > >
> > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > > > > Hi Dan
> > > > >
> > > > > Sorry for question: what do you mean for interface vnetxxxx ?
> > > > > Currently our path is :
> > > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge
-----
> > > > > vm.
> > > > >
> > > > > Which one of these ?
> > > > > Moreover, reading Fabian statements about bonding limits, today
I
> > > > > can try
> > > > to switch to a config without bonding.
> > > >
> > > > "vm" is a complicated term.
> > > >
> > > > `brctl show` would not show you a "vm" connected to a
bridge. When
> > > > you
> > > > WOULD see is a vnet888 tap device. The "other side" of this
device is
> > > > held by qemu, which implement the VM.
> > >
> > > Ok, understood and found it, vnet2
> > >
> > > >
> > > > I'm asking if the dhcp offer has reached that tap device.
> > >
> > > No, the DHCP offer packet do not reach the vnet2 interface, I can see
> > > only DHCP DISCOVER.
> >
> > Ok, so it seems that we have a problem in the host bridging.
> >
> > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> >
> > Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
> > returns to the bridge, but is not propagated to the tap device.
> > Can you suggest how to debug this further?
>
> Dump packets including the ethernet headers.
> Likely something interfered with them so the eth address is wrong.
>
> Since bonding does this sometimes, this is the most likely culprit.
We've ruled this out already - Roberto reproduces the issue without a
bond.
To me this looks like either a regression in the host side bridging. But otoh it
doesn't look
like it's happening always, because otherwise I'd expect more noise around this
issue.
- fabian
8:15 a.m.
New subject: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote:
----- Original Message -----
> On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> > On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > > > >
> > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:
> > > > > > Hi Dan
> > > > > >
> > > > > > Sorry for question: what do you mean for interface vnetxxxx
?
> > > > > > Currently our path is :
> > > > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------
bridge -----
> > > > > > vm.
> > > > > >
> > > > > > Which one of these ?
> > > > > > Moreover, reading Fabian statements about bonding limits,
today I
> > > > > > can try
> > > > > to switch to a config without bonding.
> > > > >
> > > > > "vm" is a complicated term.
> > > > >
> > > > > `brctl show` would not show you a "vm" connected to a
bridge. When
> > > > > you
> > > > > WOULD see is a vnet888 tap device. The "other side" of
this device is
> > > > > held by qemu, which implement the VM.
> > > >
> > > > Ok, understood and found it, vnet2
> > > >
> > > > >
> > > > > I'm asking if the dhcp offer has reached that tap device.
> > > >
> > > > No, the DHCP offer packet do not reach the vnet2 interface, I can
see
> > > > only DHCP DISCOVER.
> > >
> > > Ok, so it seems that we have a problem in the host bridging.
> > >
> > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> > >
> > > Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER
> > > returns to the bridge, but is not propagated to the tap device.
> > > Can you suggest how to debug this further?
> >
> > Dump packets including the ethernet headers.
> > Likely something interfered with them so the eth address is wrong.
> >
> > Since bonding does this sometimes, this is the most likely culprit.
>
> We've ruled this out already - Roberto reproduces the issue without a
> bond.
To me this looks like either a regression in the host side bridging. But otoh it
doesn't look
like it's happening always, because otherwise I'd expect more noise around this
issue.
- fabian
Hard to say. E.g. forwarding delay would do this for a while.
If eth address of the packets is okay, poke at the fbd, maybe there's
something wrong there. Maybe stp is detecting a loop - try checking that.
--
MST
1:47 a.m.
New subject: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
-----Messaggio originale-----
Da: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Per conto di
Michael S. Tsirkin
Inviato: giovedì 9 luglio 2015 15:15
A: Fabian Deutsch
Cc: users(a)ovirt.org
Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't read
DHCP offer
On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote:
> ----- Original Message -----
> > On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> > > On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > > > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > > > > >
> > > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto
wrote:
> > > > > > > Hi Dan
> > > > > > >
> > > > > > > Sorry for question: what do you mean for interface
vnetxxxx ?
> > > > > > > Currently our path is :
> > > > > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------
bridge -----
> > > > > > > vm.
> > > > > > >
> > > > > > > Which one of these ?
> > > > > > > Moreover, reading Fabian statements about bonding
limits,
today I
> > > > > > > can try
> > > > > > to switch to a config without bonding.
> > > > > >
> > > > > > "vm" is a complicated term.
> > > > > >
> > > > > > `brctl show` would not show you a "vm" connected
to a bridge.
When
> > > > > > you
> > > > > > WOULD see is a vnet888 tap device. The "other
side" of this device
is
> > > > > > held by qemu, which implement the VM.
> > > > >
> > > > > Ok, understood and found it, vnet2
> > > > >
> > > > > >
> > > > > > I'm asking if the dhcp offer has reached that tap
device.
> > > > >
> > > > > No, the DHCP offer packet do not reach the vnet2 interface, I
can see
> > > > > only DHCP DISCOVER.
> > > >
> > > > Ok, so it seems that we have a problem in the host bridging.
> > > >
> > > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> > > >
> > > > Michael, a DHCP DISCOVER is sent out of a just-booted guest, and
OFFER
> > > > returns to the bridge, but is not propagated to the tap device.
> > > > Can you suggest how to debug this further?
> > >
> > > Dump packets including the ethernet headers.
> > > Likely something interfered with them so the eth address is wrong.
> > >
> > > Since bonding does this sometimes, this is the most likely culprit.
> >
> > We've ruled this out already - Roberto reproduces the issue without a
> > bond.
>
> To me this looks like either a regression in the host side bridging. But otoh it
doesn't look
> like it's happening always, because otherwise I'd expect more noise around
this issue.
>
> - fabian
Hard to say. E.g. forwarding delay would do this for a while.
If eth address of the packets is okay, poke at the fbd, maybe there's
something wrong there. Maybe stp is detecting a loop - try checking that.
I've the tcpdump captures, let me know if are useful to analyze.
In the VLAN interface, STP=off.
RN
--
MST
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
5 a.m.
New subject: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
-----Messaggio originale-----
Da: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Per conto di
Michael S. Tsirkin
Inviato: giovedì 9 luglio 2015 15:15
A: Fabian Deutsch
Cc: users(a)ovirt.org
Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't read
DHCP offer
On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote:
> ----- Original Message -----
> > On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> > > On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > > > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > > > > >
> > > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto
wrote:
> > > > > > > Hi Dan
> > > > > > >
> > > > > > > Sorry for question: what do you mean for interface
vnetxxxx ?
> > > > > > > Currently our path is :
> > > > > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------
bridge -----
> > > > > > > vm.
> > > > > > >
> > > > > > > Which one of these ?
> > > > > > > Moreover, reading Fabian statements about bonding
limits,
today I
> > > > > > > can try
> > > > > > to switch to a config without bonding.
> > > > > >
> > > > > > "vm" is a complicated term.
> > > > > >
> > > > > > `brctl show` would not show you a "vm" connected
to a bridge.
When
> > > > > > you
> > > > > > WOULD see is a vnet888 tap device. The "other
side" of this device
is
> > > > > > held by qemu, which implement the VM.
> > > > >
> > > > > Ok, understood and found it, vnet2
> > > > >
> > > > > >
> > > > > > I'm asking if the dhcp offer has reached that tap
device.
> > > > >
> > > > > No, the DHCP offer packet do not reach the vnet2 interface, I
can see
> > > > > only DHCP DISCOVER.
> > > >
> > > > Ok, so it seems that we have a problem in the host bridging.
> > > >
> > > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> > > >
> > > > Michael, a DHCP DISCOVER is sent out of a just-booted guest, and
OFFER
> > > > returns to the bridge, but is not propagated to the tap device.
> > > > Can you suggest how to debug this further?
> > >
> > > Dump packets including the ethernet headers.
> > > Likely something interfered with them so the eth address is wrong.
> > >
> > > Since bonding does this sometimes, this is the most likely culprit.
> >
> > We've ruled this out already - Roberto reproduces the issue without a
> > bond.
>
> To me this looks like either a regression in the host side bridging. But otoh it
doesn't look
> like it's happening always, because otherwise I'd expect more noise around
this issue.
>
> - fabian
Hard to say. E.g. forwarding delay would do this for a while.
If eth address of the packets is okay, poke at the fbd, maybe there's
something wrong there. Maybe stp is detecting a loop - try checking that.
Someone is checking this ?
In tested config SPT was off.
RN
--
MST
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
5:02 a.m.
New subject: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
On Wed, Jul 29, 2015 at 12:00:38PM +0200, NUNIN Roberto wrote:
> -----Messaggio originale-----
> Da: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Per conto di
> Michael S. Tsirkin
> Inviato: giovedì 9 luglio 2015 15:15
> A: Fabian Deutsch
> Cc: users(a)ovirt.org
> Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't
read
> DHCP offer
>
> On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote:
> > ----- Original Message -----
> > > On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> > > > On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > > > > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:
> > > > > > >
> > > > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN
Roberto wrote:
> > > > > > > > Hi Dan
> > > > > > > >
> > > > > > > > Sorry for question: what do you mean for
interface vnetxxxx ?
> > > > > > > > Currently our path is :
> > > > > > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN)
------ bridge -----
> > > > > > > > vm.
> > > > > > > >
> > > > > > > > Which one of these ?
> > > > > > > > Moreover, reading Fabian statements about bonding
limits,
> today I
> > > > > > > > can try
> > > > > > > to switch to a config without bonding.
> > > > > > >
> > > > > > > "vm" is a complicated term.
> > > > > > >
> > > > > > > `brctl show` would not show you a "vm"
connected to a bridge.
> When
> > > > > > > you
> > > > > > > WOULD see is a vnet888 tap device. The "other
side" of this device
> is
> > > > > > > held by qemu, which implement the VM.
> > > > > >
> > > > > > Ok, understood and found it, vnet2
> > > > > >
> > > > > > >
> > > > > > > I'm asking if the dhcp offer has reached that tap
device.
> > > > > >
> > > > > > No, the DHCP offer packet do not reach the vnet2 interface,
I can see
> > > > > > only DHCP DISCOVER.
> > > > >
> > > > > Ok, so it seems that we have a problem in the host bridging.
> > > > >
> > > > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> > > > >
> > > > > Michael, a DHCP DISCOVER is sent out of a just-booted guest,
and
> OFFER
> > > > > returns to the bridge, but is not propagated to the tap device.
> > > > > Can you suggest how to debug this further?
> > > >
> > > > Dump packets including the ethernet headers.
> > > > Likely something interfered with them so the eth address is wrong.
> > > >
> > > > Since bonding does this sometimes, this is the most likely culprit.
> > >
> > > We've ruled this out already - Roberto reproduces the issue without a
> > > bond.
> >
> > To me this looks like either a regression in the host side bridging. But otoh
it
> doesn't look
> > like it's happening always, because otherwise I'd expect more noise
around
> this issue.
> >
> > - fabian
>
> Hard to say. E.g. forwarding delay would do this for a while.
> If eth address of the packets is okay, poke at the fbd, maybe there's
> something wrong there. Maybe stp is detecting a loop - try checking that.
Someone is checking this ?
In tested config SPT was off.
Then maybe you have a loop :)
RN
>
> --
> MST
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise private information. If you have received it in error, please
notify the sender immediately, deleting the original and all copies and destroying any
hard copies. Any other use is strictly prohibited and may be unlawful.
5:12 a.m.
New subject: R: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
-----Messaggio originale-----
Da: Michael S. Tsirkin [mailto:mst@redhat.com]
Inviato: mercoledì 29 luglio 2015 12:03
A: NUNIN Roberto
Cc: Fabian Deutsch; users(a)ovirt.org
Oggetto: Re: R: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't
read DHCP offer
On Wed, Jul 29, 2015 at 12:00:38PM +0200, NUNIN Roberto wrote:
>
> > -----Messaggio originale-----
> > Da: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Per conto
di
> > Michael S. Tsirkin
> > Inviato: giovedì 9 luglio 2015 15:15
> > A: Fabian Deutsch
> > Cc: users(a)ovirt.org
> > Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm
don't
read
> > DHCP offer
> >
> > On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote:
> > > ----- Original Message -----
> > > > On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> > > > > On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > > > > > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto
wrote:
> > > > > > > >
> > > > > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN
Roberto
wrote:
> > > > > > > > > Hi Dan
> > > > > > > > >
> > > > > > > > > Sorry for question: what do you mean for
interface vnetxxxx ?
> > > > > > > > > Currently our path is :
> > > > > > > > > eno1 - eno2 ---- bond0 ----- bond.3500
(VLAN) ------ bridge -----
> > > > > > > > > vm.
> > > > > > > > >
> > > > > > > > > Which one of these ?
> > > > > > > > > Moreover, reading Fabian statements about
bonding limits,
> > today I
> > > > > > > > > can try
> > > > > > > > to switch to a config without bonding.
> > > > > > > >
> > > > > > > > "vm" is a complicated term.
> > > > > > > >
> > > > > > > > `brctl show` would not show you a "vm"
connected to a bridge.
> > When
> > > > > > > > you
> > > > > > > > WOULD see is a vnet888 tap device. The
"other side" of this
device
> > is
> > > > > > > > held by qemu, which implement the VM.
> > > > > > >
> > > > > > > Ok, understood and found it, vnet2
> > > > > > >
> > > > > > > >
> > > > > > > > I'm asking if the dhcp offer has reached that
tap device.
> > > > > > >
> > > > > > > No, the DHCP offer packet do not reach the vnet2
interface, I can
see
> > > > > > > only DHCP DISCOVER.
> > > > > >
> > > > > > Ok, so it seems that we have a problem in the host
bridging.
> > > > > >
> > > > > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> > > > > >
> > > > > > Michael, a DHCP DISCOVER is sent out of a just-booted
guest, and
> > OFFER
> > > > > > returns to the bridge, but is not propagated to the tap
device.
> > > > > > Can you suggest how to debug this further?
> > > > >
> > > > > Dump packets including the ethernet headers.
> > > > > Likely something interfered with them so the eth address is
wrong.
> > > > >
> > > > > Since bonding does this sometimes, this is the most likely
culprit.
> > > >
> > > > We've ruled this out already - Roberto reproduces the issue
without a
> > > > bond.
> > >
> > > To me this looks like either a regression in the host side bridging. But
otoh it
> > doesn't look
> > > like it's happening always, because otherwise I'd expect more
noise
around
> > this issue.
> > >
> > > - fabian
> >
> > Hard to say. E.g. forwarding delay would do this for a while.
> > If eth address of the packets is okay, poke at the fbd, maybe there's
> > something wrong there. Maybe stp is detecting a loop - try checking that.
>
> Someone is checking this ?
> In tested config SPT was off.
Then maybe you have a loop :)
That was already checked, the MAC was unique in the VLAN.
RN
> RN
> >
> > --
> > MST
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
>
> Questo messaggio e' indirizzato esclusivamente al destinatario indicato e
potrebbe contenere informazioni confidenziali, riservate o proprietarie.
Qualora la presente venisse ricevuta per errore, si prega di segnalarlo
immediatamente al mittente, cancellando l'originale e ogni sua copia e
distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito
e potrebbe essere fonte di violazione di legge.
>
> This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information. If you have received
it in error, please notify the sender immediately, deleting the original and all
copies and destroying any hard copies. Any other use is strictly prohibited
and may be unlawful.
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
7:26 a.m.
New subject: R: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
This is a multi-part message in MIME format.
------------MIME-3653757893-462597306-delim
Content-Type: text/plain;
charset="windows-1252"
Content-Transfer-Encoding: quoted-printable
On 07/29/2015 12=3A12 PM=2C NUNIN Roberto wrote=3A
=3E=3E -----Messaggio originale-----
=3E=3E Da=3A Michael S=2E Tsirkin =5Bmailto=3Amst=40redhat=2Ecom=5D
=3E=3E Inviato=3A mercoled=EC=20=329 luglio 2015 12=3A03
=3E=3E A=3A NUNIN Roberto
=3E=3E Cc=3A Fabian Deutsch=3B users=40ovirt=2Eorg
=3E=3E Oggetto=3A Re=3A R=3A =5Bovirt-users=5D R=3A R=3A R=3A R=3A R=3A R=
=3A PXE boot of a VM on vdsm don=27t
=3E=3E read DHCP offer
=3E=3E
=3E=3E On Wed=2C Jul 29=2C 2015 at 12=3A00=3A38PM +0200=2C NUNIN Roberto wr=
ote=3A
=3E=3E=3E=3E -----Messaggio originale-----
=3E=3E=3E=3E Da=3A users-bounces=40ovirt=2Eorg =5Bmailto=3Ausers-bounces=40=
ovirt=2Eorg=5D Per conto
=3E=3E di
=3E=3E=3E=3E Michael S=2E Tsirkin
=3E=3E=3E=3E Inviato=3A gioved=EC=20=39 luglio 2015 15=3A15
=3E=3E=3E=3E A=3A Fabian Deutsch
=3E=3E=3E=3E Cc=3A users=40ovirt=2Eorg
=3E=3E=3E=3E Oggetto=3A Re=3A =5Bovirt-users=5D R=3A R=3A R=3A R=3A R=3A R=
=3A PXE boot of a VM on vdsm don=27t
=3E=3E read
=3E=3E=3E=3E DHCP offer
=3E=3E=3E=3E
=3E=3E=3E=3E On Thu=2C Jul 09=2C 2015 at 08=3A57=3A50AM -0400=2C Fabian Deu=
tsch wrote=3A
=3E=3E=3E=3E=3E ----- Original Message -----
=3E=3E=3E=3E=3E=3E On Wed=2C Jul 08=2C 2015 at 09=3A11=3A42AM +0300=2C Mich=
ael S=2E Tsirkin wrote=3A
=3E=3E=3E=3E=3E=3E=3E On Tue=2C Jul 07=2C 2015 at 05=3A13=3A28PM +0100=2C D=
an Kenigsberg wrote=3A
=3E=3E=3E=3E=3E=3E=3E=3E On Tue=2C Jul 07=2C 2015 at 10=3A14=3A54AM +0200=
=2C NUNIN Roberto wrote=3A
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E On Mon=2C Jul 06=2C 2015 at 10=3A33=3A59AM +=
0200=2C NUNIN Roberto
=3E=3E wrote=3A
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Hi Dan
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Sorry for question=3A what do you mean fo=
r interface vnetxxxx =3F
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Currently our path is =3A
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E eno1 - eno2 ---- bond0 ----- bond=2E3500=
=28VLAN=29 ------ bridge -----
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E vm=2E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Which one of these =3F
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Moreover=2C reading Fabian statements abo=
ut bonding limits=2C
=3E=3E=3E=3E today I
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E can try
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E to switch to a config without bonding=2E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =22vm=22 is a complicated term=2E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =60brctl show=60 would not show you a =22vm=
=22 connected to a bridge=2E
=3E=3E=3E=3E When
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E you
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E WOULD see is a vnet888 tap device=2E The=20=
=22other side=22 of this
=3E=3E device
=3E=3E=3E=3E is
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E held by qemu=2C which implement the VM=2E
=3E=3E=3E=3E=3E=3E=3E=3E=3E Ok=2C understood and found it=2C vnet2
=3E=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E I=27m asking if the dhcp offer has reached t=
hat tap device=2E
=3E=3E=3E=3E=3E=3E=3E=3E=3E No=2C the DHCP offer packet do not reach the vn=
et2 interface=2C I can
=3E=3E see
=3E=3E=3E=3E=3E=3E=3E=3E=3E only DHCP DISCOVER=2E
=3E=3E=3E=3E=3E=3E=3E=3E Ok=2C so it seems that we have a problem in the ho=
st bridging=2E
=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E Is it the latest kernel-3=2E10=2E0-229=2E7=2E2=2Ee=
l7=2Ex86=5F64 =3F
=3E=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E=3E Michael=2C a DHCP DISCOVER is sent out of a just-b=
ooted guest=2C and
=3E=3E=3E=3E OFFER
=3E=3E=3E=3E=3E=3E=3E=3E returns to the bridge=2C but is not propagated to=
the tap device=2E
=3E=3E=3E=3E=3E=3E=3E=3E Can you suggest how to debug this further=3F
=3E=3E=3E=3E=3E=3E=3E Dump packets including the ethernet headers=2E
=3E=3E=3E=3E=3E=3E=3E Likely something interfered with them so the eth addr=
ess is wrong=2E
=3E=3E=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E=3E=3E Since bonding does this sometimes=2C this is the most=
likely culprit=2E
=3E=3E=3E=3E=3E=3E We=27ve ruled this out already - Roberto reproduces the=
issue without a
=3E=3E=3E=3E=3E=3E bond=2E
=3E=3E=3E=3E=3E To me this looks like either a regression in the host side=
bridging=2E But
=3E=3E otoh it
=3E=3E=3E=3E doesn=27t look
=3E=3E=3E=3E=3E like it=27s happening always=2C because otherwise I=27d exp=
ect more noise
=3E=3E around
=3E=3E=3E=3E this issue=2E
=3E=3E=3E=3E=3E - fabian
=3E=3E=3E=3E Hard to say=2E E=2Eg=2E forwarding delay would do this for a w=
hile=2E
=3E=3E=3E=3E If eth address of the packets is okay=2C poke at the fbd=2C ma=
ybe there=27s
=3E=3E=3E=3E something wrong there=2E Maybe stp is detecting a loop - try c=
hecking that=2E
=3E=3E=3E Someone is checking this =3F
=3E=3E=3E In tested config SPT was off=2E
=3E=3E Then maybe you have a loop =3A=29
=3E That was already checked=2C the MAC was unique in the VLAN=2E
=3E
=3E RN
=3E
Did you also try a reboot of the VM=3F We have the same issue with foreman=
and both Libvirt and oVirt=2E On second boot PXE boots properly from DHCP=
=2E
Haven=27t had the time to investigate yet so we=27re using mostly image
based provisioning on oVirt at the moment=2E
Met vriendelijke groet=2C With kind regards=2C
Jorick Astrego
Netbulae Virtualization Experts=20
----------------
=09Tel=3A 053 20 30 270 =09info=40netbulae=2Eeu =09Staalsteden 4-3A =09KvK=
08198180
=09Fax=3A 053 20 30 271 =09www=2Enetbulae=2Eeu =097547 TA Enschede =09BTW=
NL821234584B01
----------------
------------MIME-3653757893-462597306-delim
Content-Type: text/html;
charset="utf-8"
Content-Transfer-Encoding: quoted-printable
=3Chtml=3E
=3Cbody=3E
<br <br On 07/29/2015 12:12 PM, NUNIN Roberto wrote: <br <font color=3D"#000000">>>
-----Messaggio originale----- </font><=
br <font color=3D"#000000">>> Da:
Michael S. Tsirkin [mailto:mst@<a href=
=3D"mailto:redhat.com">redhat.com</a>] </font <font
color=3D"#000000">>>>> When </font <font
color=3D"#000000">>>>>>>>>>>
I'm asking=
if the dhcp offer has reached that tap device. </font <font
color=3D"#000000">>>> In tested config SPT was
off. </fo=
nt <font color=3D"#000000">> That was
already checked, the MAC was unique i=
n the VLAN. </font Did you also try a reboot of the VM?
We have the same issue with foreman=
3;<br and both Libvirt and oVirt. On
second boot PXE boots properly from DHCP.=
3;<br <br Haven't had the time to investigate yet so we're using
mostly image <br=
based provisioning on oVirt at the moment. <br <br
<br =
=3CBR /=3E
=3CBR /=3E
=3Cb style=3D=22color=3A=23604c78=22=3E=3C/b=3E=3Cbr=3E=3Cspan style=3D=22c=
olor=3A=23604c78=3B=22=3E=3Cfont color=3D=22000000=22=3E=3Cspan style=3D=22=
mso-fareast-language=3Aen-gb=3B=22 lang=3D=22NL=22=3EMet vriendelijke groet=
=2C With kind regards=2C=3Cbr=3E=3Cbr=3E=3C/span=3EJorick Astrego=3C/font=
=3E=3C/span=3E=3Cb style=3D=22color=3A=23604c78=22=3E=3Cbr=3E=3Cbr=3ENetbul=
ae Virtualization Experts =3C/b=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3B=
border-top=3A1px solid =23ccc=3B=22=3E=3Ctable style=3D=22width=3A 522px=22=
=3E=3Ctbody=3E=3Ctr=3E=3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=
=22=3ETel=3A 053 20 30 270=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bf=
ont-size=3A 10px=22=3Einfo=40netbulae=2Eeu=3C/td=3E =3Ctd style=3D=22wid=
th=3A 130px=3Bfont-size=3A 10px=22=3EStaalsteden 4-3A=3C/td=3E =3Ctd sty=
le=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EKvK 08198180=3C/td=3E=3C/tr=
=3E=3Ctr=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EFax=
=3A 053 20 30 271=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=
=3A 10px=22=3Ewww=2Enetbulae=2Eeu=3C/td=3E =3Ctd style=3D=22width=3A 130=
px=3Bfont-size=3A 10px=22=3E7547 TA Enschede=3C/td=3E =3Ctd style=3D=22w=
idth=3A 130px=3Bfont-size=3A 10px=22=3EBTW NL821234584B01=3C/td=3E=3C/tr=3E=
=3C/tbody=3E=3C/table=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3Bborder-top=
=3A1px solid =23ccc=3B=22=3E=3CBR /=3E
=3C/body=3E
=3C/html=3E
------------MIME-3653757893-462597306-delim--
<br>
<font
color=3D"#000000">>> Inviato: mercoledì 29 luglio 2015
12:=
03 </font<br>
<font
color=3D"#000000">>> A: NUNIN Roberto </font<br>
<font
color=3D"#000000">>> Cc: Fabian Deutsch; users@<a
href=3D"mailt=
o:ovirt.org">ovirt.org</a> </font<br>
<font color=3D"#000000">>> Oggetto: Re: R:
[ovirt-users] R: R: R: R: =
R: R: PXE boot of a VM on vdsm don't </font<br>
<font color=3D"#000000">>> read DHCP
offer </font<br>
<font
color=3D"#000000">>> </font<br>
<font
color=3D"#000000">>> On Wed, Jul 29, 2015 at 12:00:38PM +0200,
=
NUNIN Roberto wrote: </font<br>
<font color=3D"#000000">>>>>
-----Messaggio originale----- =
</font<br>
<font
color=3D"#000000">>>>> Da: users-bounces@<a
href=3D"mailt=
o:ovirt.org">ovirt.org</a> [mailto:users-bounces@<a
href=3D"mailto:ovi=
rt.org">ovirt.org</a>] Per conto </font<br>
<font
color=3D"#000000">>> di </font<br>
<font
color=3D"#000000">>>>> Michael S.
Tsirkin </font<br>
<font
color=3D"#000000">>>>> Inviato: giovedì
9 luglio 201=
5 15:15 </font<br>
<font
color=3D"#000000">>>>> A: Fabian
Deutsch </font<br>
<font
color=3D"#000000">>>>> Cc: users@<a
href=3D"mailto:ovirt.=
org">ovirt.org</a> </font<br>
<font color=3D"#000000">>>>> Oggetto:
Re: [ovirt-users] R: R: R=
: R: R: R: PXE boot of a VM on vdsm don't </font<br>
<font
color=3D"#000000">>> read </font<br>
<font
color=3D"#000000">>>>> DHCP
offer </font<br>
<font
color=3D"#000000">>>>> </font<br>
<font
color=3D"#000000">>>>> On Thu, Jul 09, 2015 at
08:57:50AM=
-0400, Fabian Deutsch wrote: </font<br>
<font color=3D"#000000">>>>>>
----- Original Message -----&#=
13;</font<br>
<font
color=3D"#000000">>>>>>> On Wed,
Jul 08, 2015 at 09=
:11:42AM +0300, Michael S. Tsirkin wrote: </font<br>
<font
color=3D"#000000">>>>>>>> On
Tue, Jul 07, 2015 a=
t 05:13:28PM +0100, Dan Kenigsberg wrote: </font<br>
<font
color=3D"#000000">>>>>>>>>
On Tue, Jul 07, 20=
15 at 10:14:54AM +0200, NUNIN Roberto wrote: </font<br>
<font
color=3D"#000000">>>>>>>>>>>
On Mon, Ju=
l 06, 2015 at 10:33:59AM +0200, NUNIN Roberto </font<br>
<font
color=3D"#000000">>> wrote: </font<br>
<font
color=3D"#000000">>>>>>>>>>>>
Hi Dan=
</font<br>
<font
color=3D"#000000">>>>>>>>>>>> </=
font<br>
<font
color=3D"#000000">>>>>>>>>>>>
Sorry =
for question: what do you mean for interface vnetxxxx ? </font<br>
<font
color=3D"#000000">>>>>>>>>>>>
Curren=
tly our path is : </font<br>
<font
color=3D"#000000">>>>>>>>>>>>
eno1 -=
eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- </fon=
t<br>
<font
color=3D"#000000">>>>>>>>>>>>
vm.=
3;</font<br>
<font
color=3D"#000000">>>>>>>>>>>> </=
font<br>
<font
color=3D"#000000">>>>>>>>>>>>
Which =
one of these ? </font<br>
<font
color=3D"#000000">>>>>>>>>>>>
Moreov=
er, reading Fabian statements about bonding limits, </font<br>
<font
color=3D"#000000">>>>> today
I </font<br>
<font
color=3D"#000000">>>>>>>>>>>>
can tr=
y </font<br>
<font
color=3D"#000000">>>>>>>>>>>
to switch =
to a config without bonding. </font<br>
<font
color=3D"#000000">>>>>>>>>>> </font=
<br>
<font
color=3D"#000000">>>>>>>>>>>
"vm&q=
uot; is a complicated term. </font<br>
<font
color=3D"#000000">>>>>>>>>>> </font=
<br>
<font
color=3D"#000000">>>>>>>>>>>
`brctl sho=
w` would not show you a "vm" connected to a
bridge. </font><b=
r<br>
<font
color=3D"#000000">>>>>>>>>>>
you </=
font<br>
<font
color=3D"#000000">>>>>>>>>>>
WOULD see =
is a vnet888 tap device. The "other side" of this </font<br>
<font
color=3D"#000000">>> device </font<br>
<font
color=3D"#000000">>>>> is </font<br>
<font
color=3D"#000000">>>>>>>>>>>
held by qe=
mu, which implement the VM. </font<br>
<font
color=3D"#000000">>>>>>>>>>
Ok, understood=
and found it, vnet2 </font<br>
<font
color=3D"#000000">>>>>>>>>> </font><br=
<br>
<font
color=3D"#000000">>>>>>>>>>
No, the DHCP o=
ffer packet do not reach the vnet2 interface, I can </font<br>
<font
color=3D"#000000">>> see </font<br>
<font
color=3D"#000000">>>>>>>>>>
only DHCP DISC=
OVER. </font<br>
<font
color=3D"#000000">>>>>>>>>
Ok, so it seems th=
at we have a problem in the host bridging. </font<br>
<font
color=3D"#000000">>>>>>>>> </font<br>
<font
color=3D"#000000">>>>>>>>>
Is it the latest k=
ernel-3.10.0-229.7.2.el7.x86_64 ? </font<br>
<font
color=3D"#000000">>>>>>>>> </font<br>
<font
color=3D"#000000">>>>>>>>>
Michael, a DHCP DI=
SCOVER is sent out of a just-booted guest, and </font<br>
<font
color=3D"#000000">>>>>
OFFER </font<br>
<font
color=3D"#000000">>>>>>>>>
returns to the bri=
dge, but is not propagated to the tap device. </font<br>
<font
color=3D"#000000">>>>>>>>>
Can you suggest ho=
w to debug this further? </font<br>
<font
color=3D"#000000">>>>>>>>
Dump packets including=
the ethernet headers. </font<br>
<font
color=3D"#000000">>>>>>>>
Likely something inter=
fered with them so the eth address is wrong. </font<br>
<font
color=3D"#000000">>>>>>>> </font<br>
<font
color=3D"#000000">>>>>>>>
Since bonding does thi=
s sometimes, this is the most likely culprit. </font<br>
<font
color=3D"#000000">>>>>>> We've
ruled this out alrea=
dy - Roberto reproduces the issue without a </font<br>
<font
color=3D"#000000">>>>>>>
bond. </font<br>
<font
color=3D"#000000">>>>>> To me this looks
like either a=
regression in the host side bridging. But </font<br>
<font color=3D"#000000">>> otoh
it </font<br>
<font
color=3D"#000000">>>>> doesn't
look </font<br>
<font
color=3D"#000000">>>>>> like it's
happening always, be=
cause otherwise I'd expect more noise </font<br>
<font color=3D"#000000">>>
around </font<br>
<font
color=3D"#000000">>>>> this
issue. </font<br>
<font
color=3D"#000000">>>>>> -
fabian </font<br>
<font
color=3D"#000000">>>>> Hard to say. E.g.
forwarding delay=
would do this for a while. </font<br>
<font color=3D"#000000">>>>> If eth
address of the packets is o=
kay, poke at the fbd, maybe there's </font<br>
<font color=3D"#000000">>>>> something
wrong there. Maybe stp i=
s detecting a loop - try checking that. </font<br>
<font color=3D"#000000">>>> Someone is checking
this ? </font>=
<br<br>
<font
color=3D"#000000">>> Then maybe you have a loop
:) </font><=
br<br>
<font
color=3D"#000000">> </font<br>
<font color=3D"#000000">> RN </font<br>
<font
color=3D"#000000">> </font<br>
<br
9:17 a.m.
New subject: R: R: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
--_000_8C2CD99696B62B409EBEBFDF87C83FBF59C2364C71DENU1XMAIL02p_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
RGE6IHVzZXJzLWJvdW5jZXNAb3ZpcnQub3JnIFttYWlsdG86dXNlcnMtYm91bmNlc0BvdmlydC5v
cmddIFBlciBjb250byBkaSBKb3JpY2sgQXN0cmVnbw0KSW52aWF0bzogbWVyY29sZWTDrCAyOSBs
dWdsaW8gMjAxNSAxNDoyNg0KQTogdXNlcnNAb3ZpcnQub3JnDQpPZ2dldHRvOiBSZTogW292aXJ0
LXVzZXJzXSBSOiBSOiBSOiBSOiBSOiBSOiBSOiBSOiBQWEUgYm9vdCBvZiBhIFZNIG9uIHZkc20g
ZG9uJ3QgcmVhZCBESENQIG9mZmVyDQoNCg0KDQpPbiAwNy8yOS8yMDE1IDEyOjEyIFBNLCBOVU5J
TiBSb2JlcnRvIHdyb3RlOg0KPj4gLS0tLS1NZXNzYWdnaW8gb3JpZ2luYWxlLS0tLS0NCj4+IERh
OiBNaWNoYWVsIFMuIFRzaXJraW4gW21haWx0bzptc3RAcmVkaGF0LmNvbTxtYWlsdG86cmVkaGF0
LmNvbT5dDQo+PiBJbnZpYXRvOiBtZXJjb2xlZMOsIDI5IGx1Z2xpbyAyMDE1IDEyOjAzDQo+PiBB
OiBOVU5JTiBSb2JlcnRvDQo+PiBDYzogRmFiaWFuIERldXRzY2g7IHVzZXJzQG92aXJ0Lm9yZzxt
YWlsdG86b3ZpcnQub3JnPg0KPj4gT2dnZXR0bzogUmU6IFI6IFtvdmlydC11c2Vyc10gUjogUjog
UjogUjogUjogUjogUFhFIGJvb3Qgb2YgYSBWTSBvbiB2ZHNtIGRvbid0DQo+PiByZWFkIERIQ1Ag
b2ZmZXINCj4+DQo+PiBPbiBXZWQsIEp1bCAyOSwgMjAxNSBhdCAxMjowMDozOFBNICswMjAwLCBO
VU5JTiBSb2JlcnRvIHdyb3RlOg0KPj4+PiAtLS0tLU1lc3NhZ2dpbyBvcmlnaW5hbGUtLS0tLQ0K
Pj4+PiBEYTogdXNlcnMtYm91bmNlc0BvdmlydC5vcmc8bWFpbHRvOm92aXJ0Lm9yZz4gW21haWx0
bzp1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZzxtYWlsdG86b3ZpcnQub3JnPl0gUGVyIGNvbnRvDQo+
PiBkaQ0KPj4+PiBNaWNoYWVsIFMuIFRzaXJraW4NCj4+Pj4gSW52aWF0bzogZ2lvdmVkw6wgOSBs
dWdsaW8gMjAxNSAxNToxNQ0KPj4+PiBBOiBGYWJpYW4gRGV1dHNjaA0KPj4+PiBDYzogdXNlcnNA
b3ZpcnQub3JnPG1haWx0bzpvdmlydC5vcmc+DQo+Pj4+IE9nZ2V0dG86IFJlOiBbb3ZpcnQtdXNl
cnNdIFI6IFI6IFI6IFI6IFI6IFI6IFBYRSBib290IG9mIGEgVk0gb24gdmRzbSBkb24ndA0KPj4g
cmVhZA0KPj4+PiBESENQIG9mZmVyDQo+Pj4+DQo+Pj4+IE9uIFRodSwgSnVsIDA5LCAyMDE1IGF0
IDA4OjU3OjUwQU0gLTA0MDAsIEZhYmlhbiBEZXV0c2NoIHdyb3RlOg0KPj4+Pj4gLS0tLS0gT3Jp
Z2luYWwgTWVzc2FnZSAtLS0tLQ0KPj4+Pj4+IE9uIFdlZCwgSnVsIDA4LCAyMDE1IGF0IDA5OjEx
OjQyQU0gKzAzMDAsIE1pY2hhZWwgUy4gVHNpcmtpbiB3cm90ZToNCj4+Pj4+Pj4gT24gVHVlLCBK
dWwgMDcsIDIwMTUgYXQgMDU6MTM6MjhQTSArMDEwMCwgRGFuIEtlbmlnc2Jlcmcgd3JvdGU6DQo+
Pj4+Pj4+PiBPbiBUdWUsIEp1bCAwNywgMjAxNSBhdCAxMDoxNDo1NEFNICswMjAwLCBOVU5JTiBS
b2JlcnRvIHdyb3RlOg0KPj4+Pj4+Pj4+PiBPbiBNb24sIEp1bCAwNiwgMjAxNSBhdCAxMDozMzo1
OUFNICswMjAwLCBOVU5JTiBSb2JlcnRvDQo+PiB3cm90ZToNCj4+Pj4+Pj4+Pj4+IEhpIERhbg0K
Pj4+Pj4+Pj4+Pj4NCj4+Pj4+Pj4+Pj4+IFNvcnJ5IGZvciBxdWVzdGlvbjogd2hhdCBkbyB5b3Ug
bWVhbiBmb3IgaW50ZXJmYWNlIHZuZXR4eHh4ID8NCj4+Pj4+Pj4+Pj4+IEN1cnJlbnRseSBvdXIg
cGF0aCBpcyA6DQo+Pj4+Pj4+Pj4+PiBlbm8xIC0gZW5vMiAgLS0tLSBib25kMCAtLS0tLSBib25k
LjM1MDAgKFZMQU4pIC0tLS0tLSBicmlkZ2UgLS0tLS0NCj4+Pj4+Pj4+Pj4+IHZtLg0KPj4+Pj4+
Pj4+Pj4NCj4+Pj4+Pj4+Pj4+IFdoaWNoIG9uZSBvZiB0aGVzZSA/DQo+Pj4+Pj4+Pj4+PiBNb3Jl
b3ZlciwgcmVhZGluZyBGYWJpYW4gc3RhdGVtZW50cyBhYm91dCBib25kaW5nIGxpbWl0cywNCj4+
Pj4gdG9kYXkgSQ0KPj4+Pj4+Pj4+Pj4gY2FuIHRyeQ0KPj4+Pj4+Pj4+PiB0byBzd2l0Y2ggdG8g
YSBjb25maWcgd2l0aG91dCBib25kaW5nLg0KPj4+Pj4+Pj4+Pg0KPj4+Pj4+Pj4+PiAidm0iIGlz
IGEgY29tcGxpY2F0ZWQgdGVybS4NCj4+Pj4+Pj4+Pj4NCj4+Pj4+Pj4+Pj4gYGJyY3RsIHNob3dg
IHdvdWxkIG5vdCBzaG93IHlvdSBhICJ2bSIgY29ubmVjdGVkIHRvIGEgYnJpZGdlLg0KPj4+PiBX
aGVuDQo+Pj4+Pj4+Pj4+IHlvdQ0KPj4+Pj4+Pj4+PiBXT1VMRCBzZWUgaXMgYSB2bmV0ODg4IHRh
cCBkZXZpY2UuIFRoZSAib3RoZXIgc2lkZSIgb2YgdGhpcw0KPj4gZGV2aWNlDQo+Pj4+IGlzDQo+
Pj4+Pj4+Pj4+IGhlbGQgYnkgcWVtdSwgd2hpY2ggaW1wbGVtZW50IHRoZSBWTS4NCj4+Pj4+Pj4+
PiBPaywgdW5kZXJzdG9vZCBhbmQgZm91bmQgaXQsIHZuZXQyDQo+Pj4+Pj4+Pj4NCj4+Pj4+Pj4+
Pj4gSSdtIGFza2luZyBpZiB0aGUgZGhjcCBvZmZlciBoYXMgcmVhY2hlZCB0aGF0IHRhcCBkZXZp
Y2UuDQo+Pj4+Pj4+Pj4gTm8sIHRoZSBESENQIG9mZmVyIHBhY2tldCBkbyBub3QgcmVhY2ggdGhl
IHZuZXQyIGludGVyZmFjZSwgSSBjYW4NCj4+IHNlZQ0KPj4+Pj4+Pj4+IG9ubHkgREhDUCBESVND
T1ZFUi4NCj4+Pj4+Pj4+IE9rLCBzbyBpdCBzZWVtcyB0aGF0IHdlIGhhdmUgYSBwcm9ibGVtIGlu
IHRoZSBob3N0IGJyaWRnaW5nLg0KPj4+Pj4+Pj4NCj4+Pj4+Pj4+IElzIGl0IHRoZSBsYXRlc3Qg
a2VybmVsLTMuMTAuMC0yMjkuNy4yLmVsNy54ODZfNjQgPw0KPj4+Pj4+Pj4NCj4+Pj4+Pj4+IE1p
Y2hhZWwsIGEgREhDUCBESVNDT1ZFUiBpcyBzZW50IG91dCBvZiBhIGp1c3QtYm9vdGVkIGd1ZXN0
LCBhbmQNCj4+Pj4gT0ZGRVINCj4+Pj4+Pj4+IHJldHVybnMgdG8gdGhlIGJyaWRnZSwgYnV0IGlz
IG5vdCBwcm9wYWdhdGVkIHRvIHRoZSB0YXAgZGV2aWNlLg0KPj4+Pj4+Pj4gQ2FuIHlvdSBzdWdn
ZXN0IGhvdyB0byBkZWJ1ZyB0aGlzIGZ1cnRoZXI/DQo+Pj4+Pj4+IER1bXAgcGFja2V0cyBpbmNs
dWRpbmcgdGhlIGV0aGVybmV0IGhlYWRlcnMuDQo+Pj4+Pj4+IExpa2VseSBzb21ldGhpbmcgaW50
ZXJmZXJlZCB3aXRoIHRoZW0gc28gdGhlIGV0aCBhZGRyZXNzIGlzIHdyb25nLg0KPj4+Pj4+Pg0K
Pj4+Pj4+PiBTaW5jZSBib25kaW5nIGRvZXMgdGhpcyBzb21ldGltZXMsIHRoaXMgaXMgdGhlIG1v
c3QgbGlrZWx5IGN1bHByaXQuDQo+Pj4+Pj4gV2UndmUgcnVsZWQgdGhpcyBvdXQgYWxyZWFkeSAt
IFJvYmVydG8gcmVwcm9kdWNlcyB0aGUgaXNzdWUgd2l0aG91dCBhDQo+Pj4+Pj4gYm9uZC4NCj4+
Pj4+IFRvIG1lIHRoaXMgbG9va3MgbGlrZSBlaXRoZXIgYSByZWdyZXNzaW9uIGluIHRoZSBob3N0
IHNpZGUgYnJpZGdpbmcuIEJ1dA0KPj4gb3RvaCBpdA0KPj4+PiBkb2Vzbid0IGxvb2sNCj4+Pj4+
IGxpa2UgaXQncyBoYXBwZW5pbmcgYWx3YXlzLCBiZWNhdXNlIG90aGVyd2lzZSBJJ2QgZXhwZWN0
IG1vcmUgbm9pc2UNCj4+IGFyb3VuZA0KPj4+PiB0aGlzIGlzc3VlLg0KPj4+Pj4gLSBmYWJpYW4N
Cj4+Pj4gSGFyZCB0byBzYXkuIEUuZy4gZm9yd2FyZGluZyBkZWxheSB3b3VsZCBkbyB0aGlzIGZv
ciBhIHdoaWxlLg0KPj4+PiBJZiBldGggYWRkcmVzcyBvZiB0aGUgcGFja2V0cyBpcyBva2F5LCBw
b2tlIGF0IHRoZSBmYmQsIG1heWJlIHRoZXJlJ3MNCj4+Pj4gc29tZXRoaW5nIHdyb25nIHRoZXJl
LiBNYXliZSBzdHAgaXMgZGV0ZWN0aW5nIGEgbG9vcCAtIHRyeSBjaGVja2luZyB0aGF0Lg0KPj4+
IFNvbWVvbmUgaXMgY2hlY2tpbmcgdGhpcyA/DQo+Pj4gSW4gdGVzdGVkIGNvbmZpZyBTUFQgd2Fz
IG9mZi4NCj4+IFRoZW4gbWF5YmUgeW91IGhhdmUgYSBsb29wIDopDQo+IFRoYXQgd2FzIGFscmVh
ZHkgY2hlY2tlZCwgdGhlIE1BQyB3YXMgdW5pcXVlIGluIHRoZSBWTEFOLg0KPg0KPiBSTg0KPg0K
DQo+RGlkIHlvdSBhbHNvIHRyeSBhIHJlYm9vdCBvZiB0aGUgVk0/IFdlIGhhdmUgdGhlIHNhbWUg
aXNzdWUgd2l0aCBmb3JlbWFuDQo+YW5kIGJvdGggTGlidmlydCBhbmQgb1ZpcnQuIE9uIHNlY29u
ZCBib290IFBYRSBib290cyBwcm9wZXJseSBmcm9tIERIQ1AuDQoNClllcyB0cmllZCBtb3JlIHRo
YW4gb25lIHRpbWUgdGhlIHJlYm9vdCwgd2l0aG91dCBjaGFuZ2VzLg0KUk4NCg0KDQo+SGF2ZW4n
dCBoYWQgdGhlIHRpbWUgdG8gaW52ZXN0aWdhdGUgeWV0IHNvIHdlJ3JlIHVzaW5nIG1vc3RseSBp
bWFnZQ0KPmJhc2VkIHByb3Zpc2lvbmluZyBvbiBvVmlydCBhdCB0aGUgbW9tZW50Lg0KDQoNCg0K
DQoNCk1ldCB2cmllbmRlbGlqa2UgZ3JvZXQsIFdpdGgga2luZCByZWdhcmRzLA0KDQpKb3JpY2sg
QXN0cmVnbw0KDQpOZXRidWxhZSBWaXJ0dWFsaXphdGlvbiBFeHBlcnRzDQpfX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fXw0KVGVsOiAwNTMgMjAgMzAgMjcwDQoNCmluZm9AbmV0YnVsYWUu
ZXU8bWFpbHRvOmluZm9AbmV0YnVsYWUuZXU+DQoNClN0YWFsc3RlZGVuIDQtM0ENCg0KS3ZLIDA4
MTk4MTgwDQoNCkZheDogMDUzIDIwIDMwIDI3MQ0KDQp3d3cubmV0YnVsYWUuZXU8aHR0cDovL3d3
dy5uZXRidWxhZS5ldT4NCg0KNzU0NyBUQSBFbnNjaGVkZQ0KDQpCVFcgTkw4MjEyMzQ1ODRCMDEN
Cg0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KDQoNCl9fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fDQpRdWVzdG8gbWVzc2FnZ2lvIGUnIGluZGlyaXp6YXRvIGVzY2x1
c2l2YW1lbnRlIGFsIGRlc3RpbmF0YXJpbyBpbmRpY2F0byBlIHBvdHJlYmJlIGNvbnRlbmVyZSBp
bmZvcm1hemlvbmkgY29uZmlkZW56aWFsaSwgcmlzZXJ2YXRlIG8gcHJvcHJpZXRhcmllLiBRdWFs
b3JhIGxhIHByZXNlbnRlIHZlbmlzc2UgcmljZXZ1dGEgcGVyIGVycm9yZSwgc2kgcHJlZ2EgZGkg
c2VnbmFsYXJsbyBpbW1lZGlhdGFtZW50ZSBhbCBtaXR0ZW50ZSwgY2FuY2VsbGFuZG8gbCdvcmln
aW5hbGUgZSBvZ25pIHN1YSBjb3BpYSBlIGRpc3RydWdnZW5kbyBldmVudHVhbGkgY29waWUgY2Fy
dGFjZWUuIE9nbmkgYWx0cm8gdXNvIGUnIHN0cmV0dGFtZW50ZSBwcm9pYml0byBlIHBvdHJlYmJl
IGVzc2VyZSBmb250ZSBkaSB2aW9sYXppb25lIGRpIGxlZ2dlLg0KDQpUaGlzIG1lc3NhZ2UgaXMg
Zm9yIHRoZSBkZXNpZ25hdGVkIHJlY2lwaWVudCBvbmx5IGFuZCBtYXkgY29udGFpbiBwcml2aWxl
Z2VkLCBwcm9wcmlldGFyeSwgb3Igb3RoZXJ3aXNlIHByaXZhdGUgaW5mb3JtYXRpb24uIElmIHlv
dSBoYXZlIHJlY2VpdmVkIGl0IGluIGVycm9yLCBwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXIgaW1t
ZWRpYXRlbHksIGRlbGV0aW5nIHRoZSBvcmlnaW5hbCBhbmQgYWxsIGNvcGllcyBhbmQgZGVzdHJv
eWluZyBhbnkgaGFyZCBjb3BpZXMuIEFueSBvdGhlciB1c2UgaXMgc3RyaWN0bHkgcHJvaGliaXRl
ZCBhbmQgbWF5IGJlIHVubGF3ZnVsLg0K
--_000_8C2CD99696B62B409EBEBFDF87C83FBF59C2364C71DENU1XMAIL02p_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64
PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTQgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPCEtLVtp
ZiAhbXNvXT48c3R5bGU+dlw6KiB7YmVoYXZpb3I6dXJsKCNkZWZhdWx0I1ZNTCk7fQ0Kb1w6KiB7
YmVoYXZpb3I6dXJsKCNkZWZhdWx0I1ZNTCk7fQ0Kd1w6KiB7YmVoYXZpb3I6dXJsKCNkZWZhdWx0
I1ZNTCk7fQ0KLnNoYXBlIHtiZWhhdmlvcjp1cmwoI2RlZmF1bHQjVk1MKTt9DQo8L3N0eWxlPjwh
W2VuZGlmXS0tPjxzdHlsZT48IS0tDQovKiBGb250IERlZmluaXRpb25zICovDQpAZm9udC1mYWNl
DQoJe2ZvbnQtZmFtaWx5OiJDYW1icmlhIE1hdGgiOw0KCXBhbm9zZS0xOjIgNCA1IDMgNSA0IDYg
MyAyIDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpDYWxpYnJpOw0KCXBhbm9zZS0xOjIg
MTUgNSAyIDIgMiA0IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6IlNlZ29lIFVJ
IjsNCglwYW5vc2UtMToyIDExIDUgMiA0IDIgNCAyIDIgMzt9DQovKiBTdHlsZSBEZWZpbml0aW9u
cyAqLw0KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46
MGNtOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNpemU6MTIuMHB0Ow0KCWZvbnQt
ZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0KYTpsaW5rLCBzcGFuLk1zb0h5cGVy
bGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsNCgl0ZXh0LWRlY29y
YXRpb246dW5kZXJsaW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZA0K
CXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxlOw0KCXRleHQtZGVjb3JhdGlv
bjp1bmRlcmxpbmU7fQ0Kc3Bhbi5TdGlsZU1lc3NhZ2dpb0RpUG9zdGFFbGV0dHJvbmljYTE3DQoJ
e21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIiwi
c2Fucy1zZXJpZiI7DQoJY29sb3I6IzFGNDk3RDt9DQouTXNvQ2hwRGVmYXVsdA0KCXttc28tc3R5
bGUtdHlwZTpleHBvcnQtb25seTsNCglmb250LXNpemU6MTAuMHB0O30NCkBwYWdlIFdvcmRTZWN0
aW9uMQ0KCXtzaXplOjYxMi4wcHQgNzkyLjBwdDsNCgltYXJnaW46NzAuODVwdCAyLjBjbSAyLjBj
bSAyLjBjbTt9DQpkaXYuV29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rpb24xO30NCi0tPjwv
c3R5bGU+PCEtLVtpZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWRlZmF1bHRzIHY6ZXh0PSJl
ZGl0IiBzcGlkbWF4PSIxMDI2IiAvPg0KPC94bWw+PCFbZW5kaWZdLS0+PCEtLVtpZiBndGUgbXNv
IDldPjx4bWw+DQo8bzpzaGFwZWxheW91dCB2OmV4dD0iZWRpdCI+DQo8bzppZG1hcCB2OmV4dD0i
ZWRpdCIgZGF0YT0iMSIgLz4NCjwvbzpzaGFwZWxheW91dD48L3htbD48IVtlbmRpZl0tLT4NCjwv
aGVhZD4NCjxib2R5IGxhbmc9IklUIiBsaW5rPSJibHVlIiB2bGluaz0icHVycGxlIj4NCjxkaXYg
Y2xhc3M9IldvcmRTZWN0aW9uMSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48Yj48c3BhbiBzdHls
ZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtTZWdvZSBVSSZxdW90OywmcXVv
dDtzYW5zLXNlcmlmJnF1b3Q7Ij5EYTo8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSJmb250LXNpemU6
MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O1NlZ29lIFVJJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYm
cXVvdDsiPiB1c2Vycy1ib3VuY2VzQG92aXJ0Lm9yZyBbbWFpbHRvOnVzZXJzLWJvdW5jZXNAb3Zp
cnQub3JnXQ0KPGI+UGVyIGNvbnRvIGRpIDwvYj5Kb3JpY2sgQXN0cmVnbzxicj4NCjxiPkludmlh
dG86PC9iPiBtZXJjb2xlZMOsIDI5IGx1Z2xpbyAyMDE1IDE0OjI2PGJyPg0KPGI+QTo8L2I+IHVz
ZXJzQG92aXJ0Lm9yZzxicj4NCjxiPk9nZ2V0dG86PC9iPiBSZTogW292aXJ0LXVzZXJzXSBSOiBS
OiBSOiBSOiBSOiBSOiBSOiBSOiBQWEUgYm9vdCBvZiBhIFZNIG9uIHZkc20gZG9uJ3QgcmVhZCBE
SENQIG9mZmVyPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86
cD4mbmJzcDs8L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQo8YnI+DQpPbiAw
Ny8yOS8yMDE1IDEyOjEyIFBNLCBOVU5JTiBSb2JlcnRvIHdyb3RlOiA8YnI+DQo8c3BhbiBzdHls
ZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7IC0tLS0tTWVzc2FnZ2lvIG9yaWdpbmFsZS0tLS0tIDwv
c3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7IERhOiBNaWNoYWVs
IFMuIFRzaXJraW4gW21haWx0bzptc3RAPGEgaHJlZj0ibWFpbHRvOnJlZGhhdC5jb20iPnJlZGhh
dC5jb208L2E+XQ0KPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZn
dDsgSW52aWF0bzogbWVyY29sZWTDrCAyOSBsdWdsaW8gMjAxNSAxMjowMyA8L3NwYW4+PGJyPg0K
PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyBBOiBOVU5JTiBSb2JlcnRvIDwvc3Bh
bj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7IENjOiBGYWJpYW4gRGV1
dHNjaDsgdXNlcnNAPGEgaHJlZj0ibWFpbHRvOm92aXJ0Lm9yZyI+b3ZpcnQub3JnPC9hPg0KPC9z
cGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsgT2dnZXR0bzogUmU6
IFI6IFtvdmlydC11c2Vyc10gUjogUjogUjogUjogUjogUjogUFhFIGJvb3Qgb2YgYSBWTSBvbiB2
ZHNtIGRvbid0DQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0
OyByZWFkIERIQ1Agb2ZmZXIgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+
Jmd0OyZndDsgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsg
T24gV2VkLCBKdWwgMjksIDIwMTUgYXQgMTI6MDA6MzhQTSAmIzQzOzAyMDAsIE5VTklOIFJvYmVy
dG8gd3JvdGU6DQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0
OyZndDsmZ3Q7IC0tLS0tTWVzc2FnZ2lvIG9yaWdpbmFsZS0tLS0tIDwvc3Bhbj48YnI+DQo8c3Bh
biBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsgRGE6IHVzZXJzLWJvdW5jZXNA
PGEgaHJlZj0ibWFpbHRvOm92aXJ0Lm9yZyI+b3ZpcnQub3JnPC9hPiZuYnNwO1ttYWlsdG86dXNl
cnMtYm91bmNlc0A8YSBocmVmPSJtYWlsdG86b3ZpcnQub3JnIj5vdmlydC5vcmc8L2E+XSBQZXIg
Y29udG8NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7IGRp
IDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsg
TWljaGFlbCBTLiBUc2lya2luIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2si
PiZndDsmZ3Q7Jmd0OyZndDsgSW52aWF0bzogZ2lvdmVkw6wgOSBsdWdsaW8gMjAxNSAxNToxNSA8
L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7IEE6
IEZhYmlhbiBEZXV0c2NoIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZn
dDsmZ3Q7Jmd0OyZndDsgQ2M6IHVzZXJzQDxhIGhyZWY9Im1haWx0bzpvdmlydC5vcmciPm92aXJ0
Lm9yZzwvYT4NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7
Jmd0OyZndDsgT2dnZXR0bzogUmU6IFtvdmlydC11c2Vyc10gUjogUjogUjogUjogUjogUjogUFhF
IGJvb3Qgb2YgYSBWTSBvbiB2ZHNtIGRvbid0DQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNv
bG9yOmJsYWNrIj4mZ3Q7Jmd0OyByZWFkIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6
YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsgREhDUCBvZmZlciA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5
bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7IDwvc3Bhbj48YnI+DQo8c3BhbiBzdHls
ZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsgT24gVGh1LCBKdWwgMDksIDIwMTUgYXQg
MDg6NTc6NTBBTSAtMDQwMCwgRmFiaWFuIERldXRzY2ggd3JvdGU6DQo8L3NwYW4+PGJyPg0KPHNw
YW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyAtLS0tLSBPcmlnaW5h
bCBNZXNzYWdlIC0tLS0tIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZn
dDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBPbiBXZWQsIEp1bCAwOCwgMjAxNSBhdCAwOToxMTo0MkFN
ICYjNDM7MDMwMCwgTWljaGFlbCBTLiBUc2lya2luIHdyb3RlOg0KPC9zcGFuPjxicj4NCjxzcGFu
IHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBPbiBUdWUs
IEp1bCAwNywgMjAxNSBhdCAwNToxMzoyOFBNICYjNDM7MDEwMCwgRGFuIEtlbmlnc2Jlcmcgd3Jv
dGU6DQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsm
Z3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBPbiBUdWUsIEp1bCAwNywgMjAxNSBhdCAxMDoxNDo1NEFNICYj
NDM7MDIwMCwgTlVOSU4gUm9iZXJ0byB3cm90ZToNCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0i
Y29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgT24g
TW9uLCBKdWwgMDYsIDIwMTUgYXQgMTA6MzM6NTlBTSAmIzQzOzAyMDAsIE5VTklOIFJvYmVydG8N
Cjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7IHdyb3RlOiA8
L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBIaSBEYW4gPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxl
PSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZn
dDsgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgU29ycnkgZm9yIHF1ZXN0aW9uOiB3aGF0IGRv
IHlvdSBtZWFuIGZvciBpbnRlcmZhY2Ugdm5ldHh4eHggPw0KPC9zcGFuPjxicj4NCjxzcGFuIHN0
eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0
OyZndDsgQ3VycmVudGx5IG91ciBwYXRoIGlzIDogPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJj
b2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsg
ZW5vMSAtIGVubzImbmJzcDsgLS0tLSBib25kMCAtLS0tLSBib25kLjM1MDAgKFZMQU4pIC0tLS0t
LSBicmlkZ2UgLS0tLS0NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZn
dDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IHZtLiA8L3NwYW4+PGJy
Pg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7
Jmd0OyZndDsmZ3Q7Jmd0OyA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4m
Z3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBXaGljaCBvbmUgb2Yg
dGhlc2UgPyA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZn
dDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBNb3Jlb3ZlciwgcmVhZGluZyBGYWJp
YW4gc3RhdGVtZW50cyBhYm91dCBib25kaW5nIGxpbWl0cywNCjwvc3Bhbj48YnI+DQo8c3BhbiBz
dHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsgdG9kYXkgSSA8L3NwYW4+PGJyPg0K
PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0
OyZndDsmZ3Q7Jmd0OyBjYW4gdHJ5IDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6Ymxh
Y2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgdG8gc3dpdGNoIHRv
IGEgY29uZmlnIHdpdGhvdXQgYm9uZGluZy4gPC9zcGFuPg0KPGJyPg0KPHNwYW4gc3R5bGU9ImNv
bG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IDwvc3Bh
bj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsgJnF1b3Q7dm0mcXVvdDsgaXMgYSBjb21wbGljYXRlZCB0ZXJtLiA8
L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6Ymxh
Y2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgYGJyY3RsIHNob3dg
IHdvdWxkIG5vdCBzaG93IHlvdSBhICZxdW90O3ZtJnF1b3Q7IGNvbm5lY3RlZCB0byBhIGJyaWRn
ZS4NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZn
dDsgV2hlbiA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZn
dDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IHlvdSA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5
bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7
IFdPVUxEIHNlZSBpcyBhIHZuZXQ4ODggdGFwIGRldmljZS4gVGhlICZxdW90O290aGVyIHNpZGUm
cXVvdDsgb2YgdGhpcw0KPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0
OyZndDsgZGV2aWNlIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsm
Z3Q7Jmd0OyZndDsgaXMgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBoZWxkIGJ5IHFlbXUsIHdoaWNo
IGltcGxlbWVudCB0aGUgVk0uIDwvc3Bhbj4NCjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFj
ayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IE9rLCB1bmRlcnN0b29kIGFu
ZCBmb3VuZCBpdCwgdm5ldDIgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+
Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IDwvc3Bhbj48YnI+DQo8c3BhbiBz
dHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZn
dDsgSSdtIGFza2luZyBpZiB0aGUgZGhjcCBvZmZlciBoYXMgcmVhY2hlZCB0aGF0IHRhcCBkZXZp
Y2UuDQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsm
Z3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgTm8sIHRoZSBESENQIG9mZmVyIHBhY2tldCBkbyBub3Qg
cmVhY2ggdGhlIHZuZXQyIGludGVyZmFjZSwgSSBjYW4NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHls
ZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7IHNlZSA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNv
bG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgb25seSBESENQ
IERJU0NPVkVSLiA8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBPaywgc28gaXQgc2VlbXMgdGhhdCB3ZSBoYXZlIGEg
cHJvYmxlbSBpbiB0aGUgaG9zdCBicmlkZ2luZy4NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0i
Y29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IDwvc3Bhbj48YnI+
DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsm
Z3Q7IElzIGl0IHRoZSBsYXRlc3Qga2VybmVsLTMuMTAuMC0yMjkuNy4yLmVsNy54ODZfNjQgPw0K
PC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZn
dDsmZ3Q7Jmd0OyZndDsgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0
OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgTWljaGFlbCwgYSBESENQIERJU0NPVkVSIGlz
IHNlbnQgb3V0IG9mIGEganVzdC1ib290ZWQgZ3Vlc3QsIGFuZA0KPC9zcGFuPjxicj4NCjxzcGFu
IHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyBPRkZFUiA8L3NwYW4+PGJyPg0K
PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0
OyByZXR1cm5zIHRvIHRoZSBicmlkZ2UsIGJ1dCBpcyBub3QgcHJvcGFnYXRlZCB0byB0aGUgdGFw
IGRldmljZS4NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7
Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IENhbiB5b3Ugc3VnZ2VzdCBob3cgdG8gZGVidWcgdGhp
cyBmdXJ0aGVyPyA8L3NwYW4+DQo8YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsm
Z3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgRHVtcCBwYWNrZXRzIGluY2x1ZGluZyB0aGUgZXRoZXJu
ZXQgaGVhZGVycy4gPC9zcGFuPg0KPGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7
Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IExpa2VseSBzb21ldGhpbmcgaW50ZXJmZXJlZCB3aXRo
IHRoZW0gc28gdGhlIGV0aCBhZGRyZXNzIGlzIHdyb25nLg0KPC9zcGFuPjxicj4NCjxzcGFuIHN0
eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyA8L3NwYW4+PGJy
Pg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7
IFNpbmNlIGJvbmRpbmcgZG9lcyB0aGlzIHNvbWV0aW1lcywgdGhpcyBpcyB0aGUgbW9zdCBsaWtl
bHkgY3VscHJpdC4NCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsm
Z3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBXZSd2ZSBydWxlZCB0aGlzIG91dCBhbHJlYWR5IC0gUm9iZXJ0
byByZXByb2R1Y2VzIHRoZSBpc3N1ZSB3aXRob3V0IGENCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHls
ZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBib25kLiA8L3NwYW4+PGJy
Pg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBUbyBtZSB0
aGlzIGxvb2tzIGxpa2UgZWl0aGVyIGEgcmVncmVzc2lvbiBpbiB0aGUgaG9zdCBzaWRlIGJyaWRn
aW5nLiBCdXQNCjwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7
IG90b2ggaXQgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsm
Z3Q7Jmd0OyBkb2Vzbid0IGxvb2sgPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFj
ayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgbGlrZSBpdCdzIGhhcHBlbmluZyBhbHdheXMsIGJlY2F1
c2Ugb3RoZXJ3aXNlIEknZCBleHBlY3QgbW9yZSBub2lzZQ0KPC9zcGFuPjxicj4NCjxzcGFuIHN0
eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsgYXJvdW5kIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHls
ZT0iY29sb3I6YmxhY2siPiZndDsmZ3Q7Jmd0OyZndDsgdGhpcyBpc3N1ZS4gPC9zcGFuPjxicj4N
CjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyZndDsgLSBmYWJpYW4g
PC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7Jmd0OyBI
YXJkIHRvIHNheS4gRS5nLiBmb3J3YXJkaW5nIGRlbGF5IHdvdWxkIGRvIHRoaXMgZm9yIGEgd2hp
bGUuDQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsm
Z3Q7IElmIGV0aCBhZGRyZXNzIG9mIHRoZSBwYWNrZXRzIGlzIG9rYXksIHBva2UgYXQgdGhlIGZi
ZCwgbWF5YmUgdGhlcmUncw0KPC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+
Jmd0OyZndDsmZ3Q7Jmd0OyBzb21ldGhpbmcgd3JvbmcgdGhlcmUuIE1heWJlIHN0cCBpcyBkZXRl
Y3RpbmcgYSBsb29wIC0gdHJ5IGNoZWNraW5nIHRoYXQuDQo8L3NwYW4+PGJyPg0KPHNwYW4gc3R5
bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7Jmd0OyZndDsgU29tZW9uZSBpcyBjaGVja2luZyB0aGlzID8g
PC9zcGFuPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyZndDsmZ3Q7IEluIHRl
c3RlZCBjb25maWcgU1BUIHdhcyBvZmYuIDwvc3Bhbj48YnI+DQo8c3BhbiBzdHlsZT0iY29sb3I6
YmxhY2siPiZndDsmZ3Q7IFRoZW4gbWF5YmUgeW91IGhhdmUgYSBsb29wIDopIDwvc3Bhbj48YnI+
DQo8c3BhbiBzdHlsZT0iY29sb3I6YmxhY2siPiZndDsgVGhhdCB3YXMgYWxyZWFkeSBjaGVja2Vk
LCB0aGUgTUFDIHdhcyB1bmlxdWUgaW4gdGhlIFZMQU4uDQo8L3NwYW4+PGJyPg0KPHNwYW4gbGFu
Zz0iRU4tVVMiIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyA8L3NwYW4+PHNwYW4gbGFuZz0iRU4t
VVMiPjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Jmd0OyBSTiA8L3NwYW4+PGJyPg0K
PHNwYW4gc3R5bGU9ImNvbG9yOmJsYWNrIj4mZ3Q7IDwvc3Bhbj48YnI+DQo8YnI+DQo8c3BhbiBz
dHlsZT0iY29sb3I6IzFGNDk3RCI+Jmd0Ozwvc3Bhbj5EaWQgeW91IGFsc28gdHJ5IGEgcmVib290
IG9mIHRoZSBWTT8gV2UgaGF2ZSB0aGUgc2FtZSBpc3N1ZSB3aXRoIGZvcmVtYW4NCjxicj4NCjxz
cGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj4mZ3Q7PC9zcGFuPmFuZCBib3RoIExpYnZpcnQgYW5k
IG9WaXJ0LiA8L3NwYW4+T24gc2Vjb25kIGJvb3QgUFhFIGJvb3RzIHByb3Blcmx5IGZyb20gREhD
UC4NCjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZh
bWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFG
NDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZx
dW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj5Z
ZXMgdHJpZWQgbW9yZSB0aGFuIG9uZSB0aW1lIHRoZSByZWJvb3QsIHdpdGhvdXQgY2hhbmdlcy48
bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF
Ti1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZx
dW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPlJOPG86cD48L286cD48
L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxicj4N
Cjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj4mZ3Q7PC9zcGFuPkhhdmVuJ3QgaGFk
IHRoZSB0aW1lIHRvIGludmVzdGlnYXRlIHlldCBzbyB3ZSdyZSB1c2luZyBtb3N0bHkgaW1hZ2UN
Cjxicj4NCjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj4mZ3Q7PC9zcGFuPmJhc2VkIHByb3Zp
c2lvbmluZyBvbiBvVmlydCBhdCB0aGUgbW9tZW50LiA8YnI+DQo8YnI+DQo8YnI+DQo8YnI+DQo8
YnI+DQo8YnI+DQo8L3NwYW4+PHNwYW4gbGFuZz0iTkwiIHN0eWxlPSJjb2xvcjpibGFjazttc28t
ZmFyZWFzdC1sYW5ndWFnZTpFTi1HQiI+TWV0IHZyaWVuZGVsaWprZSBncm9ldCwgV2l0aCBraW5k
IHJlZ2FyZHMsPGJyPg0KPGJyPg0KPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjpibGFjayI+Sm9y
aWNrIEFzdHJlZ288L3NwYW4+PGI+PHNwYW4gc3R5bGU9ImNvbG9yOiM2MDRDNzgiPjxicj4NCjxi
cj4NCk5ldGJ1bGFlIFZpcnR1YWxpemF0aW9uIEV4cGVydHMgPC9zcGFuPjwvYj48bzpwPjwvbzpw
PjwvcD4NCjxkaXYgY2xhc3M9Ik1zb05vcm1hbCIgYWxpZ249ImNlbnRlciIgc3R5bGU9InRleHQt
YWxpZ246Y2VudGVyIj4NCjxociBzaXplPSIyIiB3aWR0aD0iMTAwJSIgYWxpZ249ImNlbnRlciI+
DQo8L2Rpdj4NCjx0YWJsZSBjbGFzcz0iTXNvTm9ybWFsVGFibGUiIGJvcmRlcj0iMCIgY2VsbHBh
ZGRpbmc9IjAiIHdpZHRoPSI1MjIiIHN0eWxlPSJ3aWR0aDozOTEuNXB0Ij4NCjx0Ym9keT4NCjx0
cj4NCjx0ZCB3aWR0aD0iMTMwIiBzdHlsZT0id2lkdGg6OTcuNXB0O3BhZGRpbmc6Ljc1cHQgLjc1
cHQgLjc1cHQgLjc1cHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt
c2l6ZTo3LjVwdCI+VGVsOiAwNTMgMjAgMzAgMjcwPG86cD48L286cD48L3NwYW4+PC9wPg0KPC90
ZD4NCjx0ZCB3aWR0aD0iMTMwIiBzdHlsZT0id2lkdGg6OTcuNXB0O3BhZGRpbmc6Ljc1cHQgLjc1
cHQgLjc1cHQgLjc1cHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQt
c2l6ZTo3LjVwdCI+PGEgaHJlZj0ibWFpbHRvOmluZm9AbmV0YnVsYWUuZXUiPmluZm9AbmV0YnVs
YWUuZXU8L2E+PG86cD48L286cD48L3NwYW4+PC9wPg0KPC90ZD4NCjx0ZCB3aWR0aD0iMTMwIiBz
dHlsZT0id2lkdGg6OTcuNXB0O3BhZGRpbmc6Ljc1cHQgLjc1cHQgLjc1cHQgLjc1cHQiPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTo3LjVwdCI+U3RhYWxzdGVk
ZW4gNC0zQTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvdGQ+DQo8dGQgd2lkdGg9IjEzMCIgc3R5
bGU9IndpZHRoOjk3LjVwdDtwYWRkaW5nOi43NXB0IC43NXB0IC43NXB0IC43NXB0Ij4NCjxwIGNs
YXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6Ny41cHQiPkt2SyAwODE5ODE4
MDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjwvdGQ+DQo8L3RyPg0KPHRyPg0KPHRkIHdpZHRoPSIx
MzAiIHN0eWxlPSJ3aWR0aDo5Ny41cHQ7cGFkZGluZzouNzVwdCAuNzVwdCAuNzVwdCAuNzVwdCI+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjcuNXB0Ij5GYXg6
IDA1MyAyMCAzMCAyNzE8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8L3RkPg0KPHRkIHdpZHRoPSIx
MzAiIHN0eWxlPSJ3aWR0aDo5Ny41cHQ7cGFkZGluZzouNzVwdCAuNzVwdCAuNzVwdCAuNzVwdCI+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjcuNXB0Ij48YSBo
cmVmPSJodHRwOi8vd3d3Lm5ldGJ1bGFlLmV1Ij53d3cubmV0YnVsYWUuZXU8L2E+PG86cD48L286
cD48L3NwYW4+PC9wPg0KPC90ZD4NCjx0ZCB3aWR0aD0iMTMwIiBzdHlsZT0id2lkdGg6OTcuNXB0
O3BhZGRpbmc6Ljc1cHQgLjc1cHQgLjc1cHQgLjc1cHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTo3LjVwdCI+NzU0NyBUQSBFbnNjaGVkZTxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjwvdGQ+DQo8dGQgd2lkdGg9IjEzMCIgc3R5bGU9IndpZHRoOjk3LjVwdDtw
YWRkaW5nOi43NXB0IC43NXB0IC43NXB0IC43NXB0Ij4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIHN0eWxlPSJmb250LXNpemU6Ny41cHQiPkJUVyBOTDgyMTIzNDU4NEIwMTxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjwvdGQ+DQo8L3RyPg0KPC90Ym9keT4NCjwvdGFibGU+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxkaXYgY2xhc3M9Ik1zb05vcm1hbCIg
YWxpZ249ImNlbnRlciIgc3R5bGU9InRleHQtYWxpZ246Y2VudGVyIj4NCjxociBzaXplPSIyIiB3
aWR0aD0iMTAwJSIgYWxpZ249ImNlbnRlciI+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8YnI+DQo8aHI+DQo8Zm9udCBmYWNlPSJD
b3VyaWVyIE5ldyIgY29sb3I9IkJsYWNrIiBzaXplPSIyIj5RdWVzdG8gbWVzc2FnZ2lvIGUnIGlu
ZGlyaXp6YXRvIGVzY2x1c2l2YW1lbnRlIGFsIGRlc3RpbmF0YXJpbyBpbmRpY2F0byBlIHBvdHJl
YmJlIGNvbnRlbmVyZSBpbmZvcm1hemlvbmkgY29uZmlkZW56aWFsaSwgcmlzZXJ2YXRlIG8gcHJv
cHJpZXRhcmllLiBRdWFsb3JhIGxhIHByZXNlbnRlIHZlbmlzc2UgcmljZXZ1dGEgcGVyIGVycm9y
ZSwgc2kgcHJlZ2EgZGkgc2VnbmFsYXJsbw0KIGltbWVkaWF0YW1lbnRlIGFsIG1pdHRlbnRlLCBj
YW5jZWxsYW5kbyBsJ29yaWdpbmFsZSBlIG9nbmkgc3VhIGNvcGlhIGUgZGlzdHJ1Z2dlbmRvIGV2
ZW50dWFsaSBjb3BpZSBjYXJ0YWNlZS4gT2duaSBhbHRybyB1c28gZScgc3RyZXR0YW1lbnRlIHBy
b2liaXRvIGUgcG90cmViYmUgZXNzZXJlIGZvbnRlIGRpIHZpb2xhemlvbmUgZGkgbGVnZ2UuPGJy
Pg0KPGJyPg0KVGhpcyBtZXNzYWdlIGlzIGZvciB0aGUgZGVzaWduYXRlZCByZWNpcGllbnQgb25s
eSBhbmQgbWF5IGNvbnRhaW4gcHJpdmlsZWdlZCwgcHJvcHJpZXRhcnksIG9yIG90aGVyd2lzZSBw
cml2YXRlIGluZm9ybWF0aW9uLiBJZiB5b3UgaGF2ZSByZWNlaXZlZCBpdCBpbiBlcnJvciwgcGxl
YXNlIG5vdGlmeSB0aGUgc2VuZGVyIGltbWVkaWF0ZWx5LCBkZWxldGluZyB0aGUgb3JpZ2luYWwg
YW5kIGFsbCBjb3BpZXMgYW5kIGRlc3Ryb3lpbmcgYW55IGhhcmQNCiBjb3BpZXMuIEFueSBvdGhl
ciB1c2UgaXMgc3RyaWN0bHkgcHJvaGliaXRlZCBhbmQgbWF5IGJlIHVubGF3ZnVsLjxicj4NCjwv
Zm9udD4NCjwvYm9keT4NCjwvaHRtbD4NCg==
--_000_8C2CD99696B62B409EBEBFDF87C83FBF59C2364C71DENU1XMAIL02p_--
2:51 p.m.
New subject: R: R: R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
-----Messaggio originale-----
Da: Michael S. Tsirkin [mailto:mst@redhat.com]
Inviato: mercoledì 29 luglio 2015 12:03
A: NUNIN Roberto
Cc: Fabian Deutsch; users(a)ovirt.org
Oggetto: Re: R: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't
read DHCP offer
On Wed, Jul 29, 2015 at 12:00:38PM +0200, NUNIN Roberto wrote:
>
> > -----Messaggio originale-----
> > Da: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Per conto
di
> > Michael S. Tsirkin
> > Inviato: giovedì 9 luglio 2015 15:15
> > A: Fabian Deutsch
> > Cc: users(a)ovirt.org
> > Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm
don't
read
> > DHCP offer
> >
> > On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote:
> > > ----- Original Message -----
> > > > On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:
> > > > > On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:
> > > > > > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto
wrote:
> > > > > > > >
> > > > > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN
Roberto
wrote:
> > > > > > > > > Hi Dan
> > > > > > > > >
> > > > > > > > > Sorry for question: what do you mean for
interface vnetxxxx ?
> > > > > > > > > Currently our path is :
> > > > > > > > > eno1 - eno2 ---- bond0 ----- bond.3500
(VLAN) ------ bridge -----
> > > > > > > > > vm.
> > > > > > > > >
> > > > > > > > > Which one of these ?
> > > > > > > > > Moreover, reading Fabian statements about
bonding limits,
> > today I
> > > > > > > > > can try
> > > > > > > > to switch to a config without bonding.
> > > > > > > >
> > > > > > > > "vm" is a complicated term.
> > > > > > > >
> > > > > > > > `brctl show` would not show you a "vm"
connected to a bridge.
> > When
> > > > > > > > you
> > > > > > > > WOULD see is a vnet888 tap device. The
"other side" of this
device
> > is
> > > > > > > > held by qemu, which implement the VM.
> > > > > > >
> > > > > > > Ok, understood and found it, vnet2
> > > > > > >
> > > > > > > >
> > > > > > > > I'm asking if the dhcp offer has reached that
tap device.
> > > > > > >
> > > > > > > No, the DHCP offer packet do not reach the vnet2
interface, I can
see
> > > > > > > only DHCP DISCOVER.
> > > > > >
> > > > > > Ok, so it seems that we have a problem in the host
bridging.
> > > > > >
> > > > > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?
> > > > > >
> > > > > > Michael, a DHCP DISCOVER is sent out of a just-booted
guest, and
> > OFFER
> > > > > > returns to the bridge, but is not propagated to the tap
device.
> > > > > > Can you suggest how to debug this further?
> > > > >
> > > > > Dump packets including the ethernet headers.
> > > > > Likely something interfered with them so the eth address is
wrong.
> > > > >
> > > > > Since bonding does this sometimes, this is the most likely
culprit.
> > > >
> > > > We've ruled this out already - Roberto reproduces the issue
without a
> > > > bond.
> > >
> > > To me this looks like either a regression in the host side bridging. But
otoh it
> > doesn't look
> > > like it's happening always, because otherwise I'd expect more
noise
around
> > this issue.
> > >
> > > - fabian
> >
> > Hard to say. E.g. forwarding delay would do this for a while.
> > If eth address of the packets is okay, poke at the fbd, maybe there's
> > something wrong there. Maybe stp is detecting a loop - try checking that.
>
> Someone is checking this ?
> In tested config SPT was off.
Then maybe you have a loop :)
No, it is not a loop.
I've done further tests today and finally I've defined the following conditions.
Erratic behavior is detected only within a cluster where nodes are HP Proliant BL660cGen8,
connected to Cisco Nexus 7K thru HP FEX B22 blade interconnects and Cisco Nexus 5596
switches. All nic cards are 10Gbit.
It doesn't happen with two HP Proliant DL380G5 with 10Gbit nics, connected directly to
Cisco Nexus 5548UP switches and not happen with two HP Proliant ML350eGen8 nic 1Gbit
connected to Cisco 4948 and next the same Nexus 5548UP.
All nodes are running Centos 7.1 with latest updates and all networks are configured in
the same mode, with bonding over two nic, then vlan interfaces and bridge towards VMs.
Bonding is 4 for all and works correctly with DL380 and ML350 clusters.
Well, I've tried to change the bonding mode on the BL660 cluster to mode 1 and the
issue disappear.
In all other bonding modes, it doesn't work; bridge interfaces receive DHCP offers and
do NOT reject packets, but tap interfaces aren't receiving the offer. It works only
with mode 1.
How I can investigate further ? Desiderata is to have mode 4, to aggregate available
bandwidth.
RN
> RN
> >
> > --
> > MST
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
>
> Questo messaggio e' indirizzato esclusivamente al destinatario indicato e
potrebbe contenere informazioni confidenziali, riservate o proprietarie.
Qualora la presente venisse ricevuta per errore, si prega di segnalarlo
immediatamente al mittente, cancellando l'originale e ogni sua copia e
distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito
e potrebbe essere fonte di violazione di legge.
>
> This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information. If you have received
it in error, please notify the sender immediately, deleting the original and all
copies and destroying any hard copies. Any other use is strictly prohibited
and may be unlawful.
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
5:32 a.m.
New subject: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
----- Original Message -----
Hi Dan, guys
Sorry for very late follow-up, but we had a lot of other topics to fix just
before to go back on this one.
We have tried another approach just to check if the kernel of the vdsm iso
image used to install the host could create the problem I've reported to the
list.
Now we have reinstalled the same hardware with latest CentOS 7.1, fully
updated.
Installed vdsm, then joined the oVirt cluster.
Well, we are observing the same behavior as before.
No DHCP offer is reaching the booting VM, and:
brctl showmacs <bridge_if> show us the booting vm mac-address
tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.
We have also tried to remove ANY firewall rule.
It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if
we install a vm manually and assign a static IP, it works fine.
If we switch to dhcp, the vm don't get the dynamic one.
In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP
offer.
Any further suggestion/hint ?
I've observed this behavior in bug
https://bugzilla.redhat.com/show_bug.cgi?id=1230638
We also removed all firewall rules, checked iPXE and I also saw the requests
going out, but no replies getting to the VM.
But here it sounds like it isn't specific to bonds.
After all I did not find the solution yet.
It is probably a good idea to install the oS with a static IP, and then
switch to dhcp to then use tcpdump inside the vm to see what is reaching
the inside.
- fabian
RN
> -----Messaggio originale-----
> Da: Dan Kenigsberg [mailto:danken@redhat.com]
> Inviato: lunedì 18 maggio 2015 16:14
> A: NUNIN Roberto
> Cc: users(a)ovirt.org; ibarkan(a)redhat.com
> Oggetto: Re: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP
> offer
>
> On Fri, May 08, 2015 at 03:11:25PM +0200, NUNIN Roberto wrote:
> > Hi Dan
> > Thanks for answering
> >
> >
> >
> > > Which kernel does the el7 host run? I think that Ido has seen a case
> > > where `brctl showmacs` was not populated with the VM mac, despite a
> > > packet coming out of it.
> >
> > Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't
> available within vdsm only package.
>
> Could you try upgrading to a more up-to-date
> http://mirror.centos.org/centos-
> 7/7.1.1503/updates/x86_64/Packages/kernel-3.10.0-
> 229.4.2.el7.x86_64.rpm
> ?
>
> bridge-utils is a vdsm dependency. It must exist on your host. Please
> see if the mac of the vNIC shows up on `brctl showmacs` as it should.
>
> > >
> > > Can you tcpdump and check whether the bridge propogated the DHCP
> offer
> > > to the tap device of the said VM? Does the packet generated by
> > > `ether-wake MAC-of-VM` reach the tap device?
> >
> > Yes: host "see" the broadcast :
> > 0.000000 0.0.0.0 255.255.255.255 DHCP
> > 346 DHCP
> Discover - Transaction ID 0x69267b67
> > It came from the right MAC:
> > Source: Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > And it is tagged correctly:
> > 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
> >
> > This is the offer, on the bond interface:
> > 1.012355 10.155.124.2 10.155.124.246 DHCP
> > 346 DHCP
> Offer - Transaction ID 0x69267b67
> > Layer 2 info:
> > Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst:
> Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > Tagging on the bond:
> > 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
> >
> > The tag is correctly removed when DHCP offer is forwarded over the
> bond.3500.
> > Here's the offer content, seems everything right:
> >
> > Client IP address: 0.0.0.0 (0.0.0.0)
> > Your (client) IP address: 10.155.124.246 (10.155.124.246)
> > Next server IP address: 10.155.124.223 (10.155.124.223)
> > Relay agent IP address: 10.155.124.2 (10.155.124.2)
> > Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > Client hardware address padding: 00000000000000000000
> > Server host name: 10.155.124.223
> > Boot file name: pxelinux.0
> > Magic cookie: DHCP
> >
> > Nothing of this offer appear on the VM side.
>
> But does it show on the host's bridge? on the tap device?
>
> >
> > ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host)
> > reach the VM eth0 interface:
> > 2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL
> > 116
> MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03)
> >
> > Really strange behavior.
> >
> > Roberto
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e
potrebbe contenere informazioni confidenziali, riservate o proprietarie.
Qualora la presente venisse ricevuta per errore, si prega di segnalarlo
immediatamente al mittente, cancellando l'originale e ogni sua copia e
distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente
proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged,
proprietary, or otherwise private information. If you have received it in
error, please notify the sender immediately, deleting the original and all
copies and destroying any hard copies. Any other use is strictly prohibited
and may be unlawful.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:37 a.m.
New subject: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
Thanks for answering, my considerations below.
BR,
Roberto
-----Messaggio originale-----
Da: Fabian Deutsch [mailto:fdeutsch@redhat.com]
Inviato: venerdì 3 luglio 2015 12:33
A: NUNIN Roberto
Cc: Dan Kenigsberg; users(a)ovirt.org
Oggetto: Re: [ovirt-users] R: R: R: PXE boot of a VM on vdsm don't read
DHCP offer
----- Original Message -----
> Hi Dan, guys
>
> Sorry for very late follow-up, but we had a lot of other topics to fix just
> before to go back on this one.
>
> We have tried another approach just to check if the kernel of the vdsm iso
> image used to install the host could create the problem I've reported to the
> list.
>
> Now we have reinstalled the same hardware with latest CentOS 7.1, fully
> updated.
> Installed vdsm, then joined the oVirt cluster.
>
> Well, we are observing the same behavior as before.
> No DHCP offer is reaching the booting VM, and:
>
> brctl showmacs <bridge_if> show us the booting vm mac-address
> tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.
>
> We have also tried to remove ANY firewall rule.
>
> It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if
> we install a vm manually and assign a static IP, it works fine.
> If we switch to dhcp, the vm don't get the dynamic one.
> In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP
> offer.
>
> Any further suggestion/hint ?
I've observed this behavior in bug
https://bugzilla.redhat.com/show_bug.cgi?id=1230638
We also removed all firewall rules, checked iPXE and I also saw the requests
going out, but no replies getting to the VM.
But here it sounds like it isn't specific to bonds.
After all I did not find the solution yet.
In our config, I can see the DHCP offer until the hypervisor bridge interface toward vm
It is probably a good idea to install the oS with a static IP, and then
switch to dhcp to then use tcpdump inside the vm to see what is reaching
the inside.
Already done. Vm do not acquire the IP address and, on the vm side, tcpdump shows only
requests.
At the same time, the DHCP offer s detected on the bridge if of the hypervisor.
With static IP, vm works fine.
- fabian
>
> RN
>
> > -----Messaggio originale-----
> > Da: Dan Kenigsberg [mailto:danken@redhat.com]
> > Inviato: lunedì 18 maggio 2015 16:14
> > A: NUNIN Roberto
> > Cc: users(a)ovirt.org; ibarkan(a)redhat.com
> > Oggetto: Re: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read
DHCP
> > offer
> >
> > On Fri, May 08, 2015 at 03:11:25PM +0200, NUNIN Roberto wrote:
> > > Hi Dan
> > > Thanks for answering
> > >
> > >
> > >
> > > > Which kernel does the el7 host run? I think that Ido has seen a
case
> > > > where `brctl showmacs` was not populated with the VM mac, despite
a
> > > > packet coming out of it.
> > >
> > > Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl
isn't
> > available within vdsm only package.
> >
> > Could you try upgrading to a more up-to-date
> > http://mirror.centos.org/centos-
> > 7/7.1.1503/updates/x86_64/Packages/kernel-3.10.0-
> > 229.4.2.el7.x86_64.rpm
> > ?
> >
> > bridge-utils is a vdsm dependency. It must exist on your host. Please
> > see if the mac of the vNIC shows up on `brctl showmacs` as it should.
> >
> > > >
> > > > Can you tcpdump and check whether the bridge propogated the DHCP
> > offer
> > > > to the tap device of the said VM? Does the packet generated by
> > > > `ether-wake MAC-of-VM` reach the tap device?
> > >
> > > Yes: host "see" the broadcast :
> > > 0.000000 0.0.0.0 255.255.255.255 DHCP
> > > 346 DHCP
> > Discover - Transaction ID 0x69267b67
> > > It came from the right MAC:
> > > Source: Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > > And it is tagged correctly:
> > > 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
> > >
> > > This is the offer, on the bond interface:
> > > 1.012355 10.155.124.2 10.155.124.246 DHCP
> > > 346 DHCP
> > Offer - Transaction ID 0x69267b67
> > > Layer 2 info:
> > > Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst:
> > Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > > Tagging on the bond:
> > > 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500
> > >
> > > The tag is correctly removed when DHCP offer is forwarded over the
> > bond.3500.
> > > Here's the offer content, seems everything right:
> > >
> > > Client IP address: 0.0.0.0 (0.0.0.0)
> > > Your (client) IP address: 10.155.124.246 (10.155.124.246)
> > > Next server IP address: 10.155.124.223 (10.155.124.223)
> > > Relay agent IP address: 10.155.124.2 (10.155.124.2)
> > > Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > > Client hardware address padding: 00000000000000000000
> > > Server host name: 10.155.124.223
> > > Boot file name: pxelinux.0
> > > Magic cookie: DHCP
> > >
> > > Nothing of this offer appear on the VM side.
> >
> > But does it show on the host's bridge? on the tap device?
> >
> > >
> > > ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host)
> > > reach the VM eth0 interface:
> > > 2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL
> > > 116
> > MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03)
> > >
> > > Really strange behavior.
> > >
> > > Roberto
>
> Questo messaggio e' indirizzato esclusivamente al destinatario indicato e
> potrebbe contenere informazioni confidenziali, riservate o proprietarie.
> Qualora la presente venisse ricevuta per errore, si prega di segnalarlo
> immediatamente al mittente, cancellando l'originale e ogni sua copia e
> distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente
> proibito e potrebbe essere fonte di violazione di legge.
>
> This message is for the designated recipient only and may contain
privileged,
> proprietary, or otherwise private information. If you have received it in
> error, please notify the sender immediately, deleting the original and all
> copies and destroying any hard copies. Any other use is strictly prohibited
> and may be unlawful.
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
10:03 a.m.
New subject: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
----- Original Message -----
Thanks for answering, my considerations below.
…
> I've observed this behavior in bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1230638
> We also removed all firewall rules, checked iPXE and I also saw the
> requests
> going out, but no replies getting to the VM.
> But here it sounds like it isn't specific to bonds.
> After all I did not find the solution yet.
In our config, I can see the DHCP offer until the hypervisor bridge interface
toward vm
Just to make sure, is your setup:
NIC -- bridge -- VM?
Or is a bond involved?
>
> It is probably a good idea to install the oS with a static IP, and then
> switch to dhcp to then use tcpdump inside the vm to see what is reaching
> the inside.
Already done. Vm do not acquire the IP address and, on the vm side, tcpdump
shows only requests.
At the same time, the DHCP offer s detected on the bridge if of the
hypervisor.
With static IP, vm works fine.
Yes, it only seems to be around dhcp.
If you don't have a bond involved, then please reopen the bug abaove, and we try to
find
someone to look at it.
- fabian
12:24 p.m.
New subject: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer
…
> I've observed this behavior in bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1230638
> We also removed all firewall rules, checked iPXE and I also saw the
> requests
> going out, but no replies getting to the VM.
> But here it sounds like it isn't specific to bonds.
> After all I did not find the solution yet.
In our config, I can see the DHCP offer until the hypervisor bridge interface
toward vm
Just to make sure, is your setup:
NIC -- bridge -- VM?
Or is a bond involved?
NICs (two) - BOND (type 4) - VLAN - BRIDGE - VM
>
> It is probably a good idea to install the oS with a static IP, and then
> switch to dhcp to then use tcpdump inside the vm to see what is reaching
> the inside.
Already done. Vm do not acquire the IP address and, on the vm side, tcpdump
shows only requests.
At the same time, the DHCP offer s detected on the bridge if of the
hypervisor.
With static IP, vm works fine.
Yes, it only seems to be around dhcp.
If you don't have a bond involved, then please reopen the bug abaove, and we try to
find
someone to look at it.
- fabian
Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe
contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente
venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente,
cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni
altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the
sender immediately, deleting the original and all copies and destroying any hard copies.
Any other use is strictly prohibited and may be unlawful.
3171
days inactive
3276
days old
30 comments
6 participants
participants (6)
-
Dan Kenigsberg -
Fabian Deutsch -
Jorick Astrego -
Michael S. Tsirkin -
NUNIN Roberto -
Rik Theys