Hi Rick No, DHCP is provided by an external appliance. Thanks for answering ! Roberto -----Messaggio originale----- Da: Rik Theys [mailto:Rik.Theys@esat.kuleuven.be] Inviato: giovedì 7 maggio 2015 20:46 A: NUNIN Roberto Cc: users@ovirt.org Oggetto: Re: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer Hi, Is your DHCP server a VM running on the same host? I've seem some strange issues where a VM could not obtain a DHCP lease if it was running on the same physical machine as the client. If this is the case, I can look up what I had to change, otherwise ignore it. Regards, Rik

Further info about this case:

An already installed and running VM (Centos7) with static IPv4 assignment, if changed to DHCP mode, do not acquire the IP address.

In this case, tcpdump taken on the VM, do not show DHCP offer packets that are instead seen on the host bond interface.

Seems that something is filtering DHCP offers between host physical eth interfaces and VM virtio eth interface.

Physical servers on the same VLAN keep DHCP offers and boot from PXE correctly.

Roberto

...

Hi all

...

We are using oVirt engine 3.5.1-0.0 on Centos 6.6

...

We are deploying two hosts with vdsm-4.16.10-8.gitc937927.el7.x86_64

...

No hosted-engine, it run on a dedicates VM, outside oVirt.

...

Behavior: PXE boot of a VM, ends in timeout (0x4c106035), instead to accept the DHCP offer coming from DHCP server.

...

Tcpdump capture started on the vdsm host, bond0 interface shows clearly that DHCP offers reach the vdsm interfaces three times before PXE client ends in timeout.

...

Incoming DHCP offer is correctly tagged when it comes to the bond0 interface and forwarded to the bond0.bridge interface.

...

PXE simply ignore it. PXE version is gPXE 0.9.7.

...

bond0.bridge interface is already setup with STP=off and DELAY=0.

...

If we install a VM using command line boot parameters, VM install & run fine. The issue is only related to PXE process, >when it is expected to use the DHCP offer.

...

I can provide tcpdump capture, but Iʼve not attached to the email because Iʼm quite new of the community and donʼt know if it is allowed/correct.

...

On another host, under the same engine, running vdsm-4.16.12-7.gita30da75.el6.x86_64 on Centos6.6, this behavior is not happening, everything works fine.

...

Any idea/suggestion/further investigation ?

...

Thanks for attention

...

Best regards

Roberto Nunin

Infrastructure Manager

Italy

Here are interfaces configs:

eno1:

DEVICE="eno1"

HWADDR="38:63:bb:4a:47:b0"

MASTER="bond0"

NM_CONTROLLED="no"

ONBOOT="yes"

SLAVE="yes"

eno2:

DEVICE="eno2"

HWADDR="38:63:bb:4a:47:b4"

MASTER="bond0"

NM_CONTROLLED="no"

ONBOOT="yes"

SLAVE="yes"

bond0:

BONDING_OPTS="mode=4 miimon=100"

DEVICE="bond0"

NM_CONTROLLED="no"

ONBOOT="yes"

TYPE="Bond"

bond0.3500:

DEVICE=bond0.3500

VLAN=yes

BRIDGE=DMZ3_DEV

ONBOOT=no

MTU=1500

NM_CONTROLLED=no

HOTPLUG=no

DMZ3_DEV:

DEVICE=DMZ3_DEV

TYPE=Bridge

DELAY=0

STP=off

ONBOOT=no

MTU=1500

DEFROUTE=no

NM_CONTROLLED=no

HOTPLUG=no

_______________________________________________________________________________________________________________________ Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Hi Dan Thanks for answering

Which kernel does the el7 host run? I think that Ido has seen a case where `brctl showmacs` was not populated with the VM mac, despite a packet coming out of it.

Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't available within vdsm only package.

Can you tcpdump and check whether the bridge propogated the DHCP offer to the tap device of the said VM? Does the packet generated by `ether-wake MAC-of-VM` reach the tap device?

Yes: host "see" the broadcast : 0.000000 0.0.0.0 255.255.255.255 DHCP 346 DHCP Discover - Transaction ID 0x69267b67 It came from the right MAC: Source: Qumranet_15:81:03 (00:1a:4a:15:81:03) And it is tagged correctly: 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500 This is the offer, on the bond interface: 1.012355 10.155.124.2 10.155.124.246 DHCP 346 DHCP Offer - Transaction ID 0x69267b67 Layer 2 info: Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst: Qumranet_15:81:03 (00:1a:4a:15:81:03) Tagging on the bond: 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500 The tag is correctly removed when DHCP offer is forwarded over the bond.3500. Here's the offer content, seems everything right: Client IP address: 0.0.0.0 (0.0.0.0) Your (client) IP address: 10.155.124.246 (10.155.124.246) Next server IP address: 10.155.124.223 (10.155.124.223) Relay agent IP address: 10.155.124.2 (10.155.124.2) Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03) Client hardware address padding: 00000000000000000000 Server host name: 10.155.124.223 Boot file name: pxelinux.0 Magic cookie: DHCP Nothing of this offer appear on the VM side. ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host) reach the VM eth0 interface: 2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL 116 MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03) Really strange behavior. Roberto Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Hi Dan, guys Sorry for very late follow-up, but we had a lot of other topics to fix just before to go back on this one. We have tried another approach just to check if the kernel of the vdsm iso image used to install the host could create the problem I've reported to the list. Now we have reinstalled the same hardware with latest CentOS 7.1, fully updated. Installed vdsm, then joined the oVirt cluster. Well, we are observing the same behavior as before. No DHCP offer is reaching the booting VM, and: brctl showmacs <bridge_if> show us the booting vm mac-address tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server. We have also tried to remove ANY firewall rule. It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if we install a vm manually and assign a static IP, it works fine. If we switch to dhcp, the vm don't get the dynamic one. In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP offer. Any further suggestion/hint ? RN

-----Messaggio originale----- Da: Dan Kenigsberg [mailto:danken@redhat.com] Inviato: lunedì 18 maggio 2015 16:14 A: NUNIN Roberto Cc: users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer

On Fri, May 08, 2015 at 03:11:25PM +0200, NUNIN Roberto wrote:

...

Hi Dan Thanks for answering

...

Which kernel does the el7 host run? I think that Ido has seen a case where `brctl showmacs` was not populated with the VM mac, despite a packet coming out of it.

Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't available within vdsm only package.

Could you try upgrading to a more up-to-date http://mirror.centos.org/centos- 7/7.1.1503/updates/x86_64/Packages/kernel-3.10.0- 229.4.2.el7.x86_64.rpm ?

bridge-utils is a vdsm dependency. It must exist on your host. Please see if the mac of the vNIC shows up on `brctl showmacs` as it should.

...

...

Can you tcpdump and check whether the bridge propogated the DHCP

offer

...

to the tap device of the said VM? Does the packet generated by `ether-wake MAC-of-VM` reach the tap device?

Yes: host "see" the broadcast : 0.000000 0.0.0.0 255.255.255.255 DHCP 346 DHCP Discover - Transaction ID 0x69267b67 It came from the right MAC: Source: Qumranet_15:81:03 (00:1a:4a:15:81:03) And it is tagged correctly: 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500

This is the offer, on the bond interface: 1.012355 10.155.124.2 10.155.124.246 DHCP 346 DHCP Offer - Transaction ID 0x69267b67 Layer 2 info: Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst: Qumranet_15:81:03 (00:1a:4a:15:81:03) Tagging on the bond: 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500

The tag is correctly removed when DHCP offer is forwarded over the bond.3500. Here's the offer content, seems everything right:

Client IP address: 0.0.0.0 (0.0.0.0) Your (client) IP address: 10.155.124.246 (10.155.124.246) Next server IP address: 10.155.124.223 (10.155.124.223) Relay agent IP address: 10.155.124.2 (10.155.124.2) Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03) Client hardware address padding: 00000000000000000000 Server host name: 10.155.124.223 Boot file name: pxelinux.0 Magic cookie: DHCP

Nothing of this offer appear on the VM side.

But does it show on the host's bridge? on the tap device?

...

ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host) reach the VM eth0 interface: 2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL 116

MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03)

...

Really strange behavior.

Roberto

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Thanks for answering. My considerations below. BR, ROberto

-----Messaggio originale----- Da: Dan Kenigsberg [mailto:danken@redhat.com] Inviato: venerdì 3 luglio 2015 12:31 A: NUNIN Roberto Cc: users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer

...

Hi Dan, guys

Sorry for very late follow-up, but we had a lot of other topics to fix just before to go back on this one.

We have tried another approach just to check if the kernel of the vdsm iso image used to install the host could create the problem I've reported to the

On Fri, Jul 03, 2015 at 10:51:52AM +0200, NUNIN Roberto wrote: list.

...

Now we have reinstalled the same hardware with latest CentOS 7.1, fully

updated.

...

Installed vdsm, then joined the oVirt cluster.

Well, we are observing the same behavior as before. No DHCP offer is reaching the booting VM, and:

brctl showmacs <bridge_if> show us the booting vm mac-address tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.

The offer should be replicated to the tap device (vnetXXX). I assume that it does not?

The DHCP offer package coming from DHCP server flows thru the bond0 device (tagged), the VLAN device (tagged) and the bridge interface (not anymore tagged, correct), so it seems that the path back is complete, considering that brctl showmacs shows the vm mac and If we put a static IP into the vm nic, it works correctly.

...

We have also tried to remove ANY firewall rule.

It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if we

install a vm manually and assign a static IP, it works fine.

...

If we switch to dhcp, the vm don't get the dynamic one. In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP offer.

Any further suggestion/hint ?

We have mode 4, but already checked active/backup, mode 2, without changes.

Which bond mode are you using? I recently seen

https://bugzilla.redhat.com/show_bug.cgi?id=1230638#c22

closed as duplicate of

Bug 1094842 - Bonding modes 0, 5 and 6 should be avoided for VM networks

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Hi Dan Sorry for question: what do you mean for interface vnetxxxx ? Currently our path is : eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm. Which one of these ? Moreover, reading Fabian statements about bonding limits, today I can try to switch to a config without bonding. BR Roberto

-----Messaggio originale----- Da: Dan Kenigsberg [mailto:danken@redhat.com] Inviato: lunedì 6 luglio 2015 10:02 A: NUNIN Roberto Cc: users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer

...

Thanks for answering. My considerations below.

BR,

ROberto

...

-----Messaggio originale----- Da: Dan Kenigsberg [mailto:danken@redhat.com] Inviato: venerdì 3 luglio 2015 12:31 A: NUNIN Roberto Cc: users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer

On Fri, Jul 03, 2015 at 10:51:52AM +0200, NUNIN Roberto wrote:

...

Hi Dan, guys

Sorry for very late follow-up, but we had a lot of other topics to fix just before to go back on this one.

We have tried another approach just to check if the kernel of the vdsm iso image used to install the host could create the problem I've reported to

On Fri, Jul 03, 2015 at 04:32:50PM +0200, NUNIN Roberto wrote: the

...

...

list.

...

Now we have reinstalled the same hardware with latest CentOS 7.1,

fully updated.

...

Installed vdsm, then joined the oVirt cluster.

Well, we are observing the same behavior as before. No DHCP offer is reaching the booting VM, and:

brctl showmacs <bridge_if> show us the booting vm mac-address tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.

The offer should be replicated to the tap device (vnetXXX). I assume that it does not?

The DHCP offer package coming from DHCP server flows thru the bond0 device (tagged), the VLAN device (tagged) and the bridge interface (not anymore tagged, correct), so it seems that the path back is complete, considering that brctl showmacs shows the vm mac and If we put a static IP into the vm nic, it works correctly.

Let's try to trace the offer for one more link. If it does not get to the tap device vnetXXX - it is clearly a host-side issue. Maybe a bridge module bug. If you spot the offer on the tap, the it becomes more likely that the issue is in qemu or elsewere in the virt stack.

I would appreciate your assistence!

...

...

...

We have also tried to remove ANY firewall rule.

It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if

we

...

...

install a vm manually and assign a static IP, it works fine.

...

If we switch to dhcp, the vm don't get the dynamic one. In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP offer.

Any further suggestion/hint ?

We have mode 4, but already checked active/backup, mode 2, without changes.

Thanks, so it's not that bonding issue.

Regards, Dan.

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:

...

Hi Dan

Sorry for question: what do you mean for interface vnetxxxx ? Currently our path is : eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.

Which one of these ? Moreover, reading Fabian statements about bonding limits, today I can try to switch to a config without bonding.

"vm" is a complicated term.

`brctl show` would not show you a "vm" connected to a bridge. When you WOULD see is a vnet888 tap device. The "other side" of this device is held by qemu, which implement the VM.

Ok, understood and found it, vnet2

I'm asking if the dhcp offer has reached that tap device.

No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER. Roberto Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

-----Messaggio originale----- Da: Dan Kenigsberg [mailto:danken@redhat.com] Inviato: martedì 7 luglio 2015 18:13 A: NUNIN Roberto; mst@redhat.com Cc: users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: R: R: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer

...

...

On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:

...

Hi Dan

Sorry for question: what do you mean for interface vnetxxxx ? Currently our path is : eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.

Which one of these ? Moreover, reading Fabian statements about bonding limits, today I can

On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote: try

...

...

to switch to a config without bonding.

"vm" is a complicated term.

`brctl show` would not show you a "vm" connected to a bridge. When you WOULD see is a vnet888 tap device. The "other side" of this device is held by qemu, which implement the VM.

Ok, understood and found it, vnet2

...

I'm asking if the dhcp offer has reached that tap device.

No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER.

Ok, so it seems that we have a problem in the host bridging.

Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?

It is 3.10.0-229.7.2.el7.x86_64 on the host with installed Centos7.1, update, then vdsm It is 3.10.0-123.20.1.el7.x86_64 on the host installed with vdsm iso Both hosts show the issue

Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER returns to the bridge, but is not propagated to the tap device. Can you suggest how to debug this further?

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

-----Messaggio originale----- Da: Michael S. Tsirkin [mailto:mst@redhat.com] Inviato: mercoledì 8 luglio 2015 08:12 A: Dan Kenigsberg Cc: NUNIN Roberto; users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: R: R: R: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read DHCP offer

On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:

...

On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:

...

...

On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:

...

Hi Dan

Sorry for question: what do you mean for interface vnetxxxx ? Currently our path is : eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.

Which one of these ? Moreover, reading Fabian statements about bonding limits, today I

can try

...

to switch to a config without bonding.

"vm" is a complicated term.

`brctl show` would not show you a "vm" connected to a bridge. When you WOULD see is a vnet888 tap device. The "other side" of this device is held by qemu, which implement the VM.

Ok, understood and found it, vnet2

...

I'm asking if the dhcp offer has reached that tap device.

No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER.

Ok, so it seems that we have a problem in the host bridging.

Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?

Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER returns to the bridge, but is not propagated to the tap device. Can you suggest how to debug this further?

Dump packets including the ethernet headers.

I have both tcpdump capture, on the bridge interface and on the vnet interface. How I can upload output files ? Sorry for probably obvious question, first time for me :) Roberto Nunin

Likely something interfered with them so the eth address is wrong.

Since bonding does this sometimes, this is the most likely culprit.

-- MST

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

----- Original Message -----

On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:

...

On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:

...

On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:

...

...

On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote:

...

Hi Dan

Sorry for question: what do you mean for interface vnetxxxx ? Currently our path is : eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- vm.

Which one of these ? Moreover, reading Fabian statements about bonding limits, today I can try to switch to a config without bonding.

"vm" is a complicated term.

`brctl show` would not show you a "vm" connected to a bridge. When you WOULD see is a vnet888 tap device. The "other side" of this device is held by qemu, which implement the VM.

Ok, understood and found it, vnet2

...

I'm asking if the dhcp offer has reached that tap device.

No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER.

Ok, so it seems that we have a problem in the host bridging.

Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?

Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER returns to the bridge, but is not propagated to the tap device. Can you suggest how to debug this further?

Dump packets including the ethernet headers. Likely something interfered with them so the eth address is wrong.

Since bonding does this sometimes, this is the most likely culprit.

We've ruled this out already - Roberto reproduces the issue without a bond.

To me this looks like either a regression in the host side bridging. But otoh it doesn't look like it's happening always, because otherwise I'd expect more noise around this issue. - fabian

-----Messaggio originale----- Da: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Per conto di Michael S. Tsirkin Inviato: giovedì 9 luglio 2015 15:15 A: Fabian Deutsch Cc: users@ovirt.org Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer

...

----- Original Message -----

...

On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:

...

On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:

...

On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote:

...

> > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote: > > Hi Dan > > > > Sorry for question: what do you mean for interface vnetxxxx ? > > Currently our path is : > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- > > vm. > > > > Which one of these ? > > Moreover, reading Fabian statements about bonding limits, today I > > can try > to switch to a config without bonding. > > "vm" is a complicated term. > > `brctl show` would not show you a "vm" connected to a bridge. When > you > WOULD see is a vnet888 tap device. The "other side" of this device is > held by qemu, which implement the VM.

Ok, understood and found it, vnet2

> > I'm asking if the dhcp offer has reached that tap device.

No, the DHCP offer packet do not reach the vnet2 interface, I can see only DHCP DISCOVER.

Ok, so it seems that we have a problem in the host bridging.

Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?

Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER returns to the bridge, but is not propagated to the tap device. Can you suggest how to debug this further?

Dump packets including the ethernet headers. Likely something interfered with them so the eth address is wrong.

Since bonding does this sometimes, this is the most likely culprit.

We've ruled this out already - Roberto reproduces the issue without a bond.

To me this looks like either a regression in the host side bridging. But otoh it doesn't look like it's happening always, because otherwise I'd expect more noise around

On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote: this issue.

...

- fabian

Hard to say. E.g. forwarding delay would do this for a while. If eth address of the packets is okay, poke at the fbd, maybe there's something wrong there. Maybe stp is detecting a loop - try checking that.

I've the tcpdump captures, let me know if are useful to analyze. In the VLAN interface, STP=off. RN

-- MST _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

On Wed, Jul 29, 2015 at 12:00:38PM +0200, NUNIN Roberto wrote:

...

-----Messaggio originale----- Da: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Per conto di Michael S. Tsirkin Inviato: giovedì 9 luglio 2015 15:15 A: Fabian Deutsch Cc: users@ovirt.org Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer

...

----- Original Message -----

...

On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:

...

On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote:

...

On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote: > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote: > > > Hi Dan > > > > > > Sorry for question: what do you mean for interface vnetxxxx ? > > > Currently our path is : > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- > > > vm. > > > > > > Which one of these ? > > > Moreover, reading Fabian statements about bonding limits, today I > > > can try > > to switch to a config without bonding. > > > > "vm" is a complicated term. > > > > `brctl show` would not show you a "vm" connected to a bridge. When > > you > > WOULD see is a vnet888 tap device. The "other side" of this device is > > held by qemu, which implement the VM. > > Ok, understood and found it, vnet2 > > > > > I'm asking if the dhcp offer has reached that tap device. > > No, the DHCP offer packet do not reach the vnet2 interface, I can see > only DHCP DISCOVER.

Ok, so it seems that we have a problem in the host bridging.

Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ?

Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER returns to the bridge, but is not propagated to the tap device. Can you suggest how to debug this further?

Dump packets including the ethernet headers. Likely something interfered with them so the eth address is wrong.

Since bonding does this sometimes, this is the most likely culprit.

We've ruled this out already - Roberto reproduces the issue without a bond.

To me this looks like either a regression in the host side bridging. But otoh it doesn't look like it's happening always, because otherwise I'd expect more noise around

On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote: this issue.

...

- fabian

Hard to say. E.g. forwarding delay would do this for a while. If eth address of the packets is okay, poke at the fbd, maybe there's something wrong there. Maybe stp is detecting a loop - try checking that.

Someone is checking this ? In tested config SPT was off.

Then maybe you have a loop :)

RN

...

-- MST _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

This is a multi-part message in MIME format. ------------MIME-3653757893-462597306-delim Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable On 07/29/2015 12=3A12 PM=2C NUNIN Roberto wrote=3A =3E=3E -----Messaggio originale----- =3E=3E Da=3A Michael S=2E Tsirkin =5Bmailto=3Amst=40redhat=2Ecom=5D =3E=3E Inviato=3A mercoled=EC=20=329 luglio 2015 12=3A03 =3E=3E A=3A NUNIN Roberto =3E=3E Cc=3A Fabian Deutsch=3B users=40ovirt=2Eorg =3E=3E Oggetto=3A Re=3A R=3A =5Bovirt-users=5D R=3A R=3A R=3A R=3A R=3A R= =3A PXE boot of a VM on vdsm don=27t =3E=3E read DHCP offer =3E=3E =3E=3E On Wed=2C Jul 29=2C 2015 at 12=3A00=3A38PM +0200=2C NUNIN Roberto wr= ote=3A =3E=3E=3E=3E -----Messaggio originale----- =3E=3E=3E=3E Da=3A users-bounces=40ovirt=2Eorg =5Bmailto=3Ausers-bounces=40= ovirt=2Eorg=5D Per conto =3E=3E di =3E=3E=3E=3E Michael S=2E Tsirkin =3E=3E=3E=3E Inviato=3A gioved=EC=20=39 luglio 2015 15=3A15 =3E=3E=3E=3E A=3A Fabian Deutsch =3E=3E=3E=3E Cc=3A users=40ovirt=2Eorg =3E=3E=3E=3E Oggetto=3A Re=3A =5Bovirt-users=5D R=3A R=3A R=3A R=3A R=3A R= =3A PXE boot of a VM on vdsm don=27t =3E=3E read =3E=3E=3E=3E DHCP offer =3E=3E=3E=3E =3E=3E=3E=3E On Thu=2C Jul 09=2C 2015 at 08=3A57=3A50AM -0400=2C Fabian Deu= tsch wrote=3A =3E=3E=3E=3E=3E ----- Original Message ----- =3E=3E=3E=3E=3E=3E On Wed=2C Jul 08=2C 2015 at 09=3A11=3A42AM +0300=2C Mich= ael S=2E Tsirkin wrote=3A =3E=3E=3E=3E=3E=3E=3E On Tue=2C Jul 07=2C 2015 at 05=3A13=3A28PM +0100=2C D= an Kenigsberg wrote=3A =3E=3E=3E=3E=3E=3E=3E=3E On Tue=2C Jul 07=2C 2015 at 10=3A14=3A54AM +0200= =2C NUNIN Roberto wrote=3A =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E On Mon=2C Jul 06=2C 2015 at 10=3A33=3A59AM += 0200=2C NUNIN Roberto =3E=3E wrote=3A =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Hi Dan =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Sorry for question=3A what do you mean fo= r interface vnetxxxx =3F =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Currently our path is =3A =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E eno1 - eno2 ---- bond0 ----- bond=2E3500= =28VLAN=29 ------ bridge ----- =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E vm=2E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Which one of these =3F =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E Moreover=2C reading Fabian statements abo= ut bonding limits=2C =3E=3E=3E=3E today I =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E=3E can try =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E to switch to a config without bonding=2E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =22vm=22 is a complicated term=2E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E =60brctl show=60 would not show you a =22vm= =22 connected to a bridge=2E =3E=3E=3E=3E When =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E you =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E WOULD see is a vnet888 tap device=2E The=20= =22other side=22 of this =3E=3E device =3E=3E=3E=3E is =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E held by qemu=2C which implement the VM=2E =3E=3E=3E=3E=3E=3E=3E=3E=3E Ok=2C understood and found it=2C vnet2 =3E=3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E=3E=3E I=27m asking if the dhcp offer has reached t= hat tap device=2E =3E=3E=3E=3E=3E=3E=3E=3E=3E No=2C the DHCP offer packet do not reach the vn= et2 interface=2C I can =3E=3E see =3E=3E=3E=3E=3E=3E=3E=3E=3E only DHCP DISCOVER=2E =3E=3E=3E=3E=3E=3E=3E=3E Ok=2C so it seems that we have a problem in the ho= st bridging=2E =3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E Is it the latest kernel-3=2E10=2E0-229=2E7=2E2=2Ee= l7=2Ex86=5F64 =3F =3E=3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E=3E Michael=2C a DHCP DISCOVER is sent out of a just-b= ooted guest=2C and =3E=3E=3E=3E OFFER =3E=3E=3E=3E=3E=3E=3E=3E returns to the bridge=2C but is not propagated to= the tap device=2E =3E=3E=3E=3E=3E=3E=3E=3E Can you suggest how to debug this further=3F =3E=3E=3E=3E=3E=3E=3E Dump packets including the ethernet headers=2E =3E=3E=3E=3E=3E=3E=3E Likely something interfered with them so the eth addr= ess is wrong=2E =3E=3E=3E=3E=3E=3E=3E =3E=3E=3E=3E=3E=3E=3E Since bonding does this sometimes=2C this is the most= likely culprit=2E =3E=3E=3E=3E=3E=3E We=27ve ruled this out already - Roberto reproduces the= issue without a =3E=3E=3E=3E=3E=3E bond=2E =3E=3E=3E=3E=3E To me this looks like either a regression in the host side= bridging=2E But =3E=3E otoh it =3E=3E=3E=3E doesn=27t look =3E=3E=3E=3E=3E like it=27s happening always=2C because otherwise I=27d exp= ect more noise =3E=3E around =3E=3E=3E=3E this issue=2E =3E=3E=3E=3E=3E - fabian =3E=3E=3E=3E Hard to say=2E E=2Eg=2E forwarding delay would do this for a w= hile=2E =3E=3E=3E=3E If eth address of the packets is okay=2C poke at the fbd=2C ma= ybe there=27s =3E=3E=3E=3E something wrong there=2E Maybe stp is detecting a loop - try c= hecking that=2E =3E=3E=3E Someone is checking this =3F =3E=3E=3E In tested config SPT was off=2E =3E=3E Then maybe you have a loop =3A=29 =3E That was already checked=2C the MAC was unique in the VLAN=2E =3E =3E RN =3E Did you also try a reboot of the VM=3F We have the same issue with foreman= and both Libvirt and oVirt=2E On second boot PXE boots properly from DHCP= =2E Haven=27t had the time to investigate yet so we=27re using mostly image based provisioning on oVirt at the moment=2E Met vriendelijke groet=2C With kind regards=2C Jorick Astrego Netbulae Virtualization Experts=20 ---------------- =09Tel=3A 053 20 30 270 =09info=40netbulae=2Eeu =09Staalsteden 4-3A =09KvK= 08198180 =09Fax=3A 053 20 30 271 =09www=2Enetbulae=2Eeu =097547 TA Enschede =09BTW= NL821234584B01 ---------------- ------------MIME-3653757893-462597306-delim Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable =3Chtml=3E =3Cbody=3E <br> <br> On 07/29/2015 12:12 PM, NUNIN Roberto wrote: <br> <font color=3D"#000000">>> -----Messaggio originale----- </font><= br> <font color=3D"#000000">>> Da: Michael S. Tsirkin [mailto:mst@<a href= =3D"mailto:redhat.com">redhat.com</a>] </font><br> <font color=3D"#000000">>> Inviato: mercoledì 29 luglio 2015 12:= 03 </font><br> <font color=3D"#000000">>> A: NUNIN Roberto </font><br> <font color=3D"#000000">>> Cc: Fabian Deutsch; users@<a href=3D"mailt= o:ovirt.org">ovirt.org</a> </font><br> <font color=3D"#000000">>> Oggetto: Re: R: [ovirt-users] R: R: R: R: = R: R: PXE boot of a VM on vdsm don't </font><br> <font color=3D"#000000">>> read DHCP offer </font><br> <font color=3D"#000000">>> </font><br> <font color=3D"#000000">>> On Wed, Jul 29, 2015 at 12:00:38PM +0200, = NUNIN Roberto wrote: </font><br> <font color=3D"#000000">>>>> -----Messaggio originale----- = </font><br> <font color=3D"#000000">>>>> Da: users-bounces@<a href=3D"mailt= o:ovirt.org">ovirt.org</a> [mailto:users-bounces@<a href=3D"mailto:ovi= rt.org">ovirt.org</a>] Per conto </font><br> <font color=3D"#000000">>> di </font><br> <font color=3D"#000000">>>>> Michael S. Tsirkin </font><br> <font color=3D"#000000">>>>> Inviato: giovedì 9 luglio 201= 5 15:15 </font><br> <font color=3D"#000000">>>>> A: Fabian Deutsch </font><br> <font color=3D"#000000">>>>> Cc: users@<a href=3D"mailto:ovirt.= org">ovirt.org</a> </font><br> <font color=3D"#000000">>>>> Oggetto: Re: [ovirt-users] R: R: R= : R: R: R: PXE boot of a VM on vdsm don't </font><br> <font color=3D"#000000">>> read </font><br> <font color=3D"#000000">>>>> DHCP offer </font><br> <font color=3D"#000000">>>>> </font><br> <font color=3D"#000000">>>>> On Thu, Jul 09, 2015 at 08:57:50AM= -0400, Fabian Deutsch wrote: </font><br> <font color=3D"#000000">>>>>> ----- Original Message -----&#= 13;</font><br> <font color=3D"#000000">>>>>>> On Wed, Jul 08, 2015 at 09= :11:42AM +0300, Michael S. Tsirkin wrote: </font><br> <font color=3D"#000000">>>>>>>> On Tue, Jul 07, 2015 a= t 05:13:28PM +0100, Dan Kenigsberg wrote: </font><br> <font color=3D"#000000">>>>>>>>> On Tue, Jul 07, 20= 15 at 10:14:54AM +0200, NUNIN Roberto wrote: </font><br> <font color=3D"#000000">>>>>>>>>>> On Mon, Ju= l 06, 2015 at 10:33:59AM +0200, NUNIN Roberto </font><br> <font color=3D"#000000">>> wrote: </font><br> <font color=3D"#000000">>>>>>>>>>>> Hi Dan= </font><br> <font color=3D"#000000">>>>>>>>>>>> </= font><br> <font color=3D"#000000">>>>>>>>>>>> Sorry = for question: what do you mean for interface vnetxxxx ? </font><br> <font color=3D"#000000">>>>>>>>>>>> Curren= tly our path is : </font><br> <font color=3D"#000000">>>>>>>>>>>> eno1 -= eno2  ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- </fon= t><br> <font color=3D"#000000">>>>>>>>>>>> vm.
= 3;</font><br> <font color=3D"#000000">>>>>>>>>>>> </= font><br> <font color=3D"#000000">>>>>>>>>>>> Which = one of these ? </font><br> <font color=3D"#000000">>>>>>>>>>>> Moreov= er, reading Fabian statements about bonding limits, </font><br> <font color=3D"#000000">>>>> today I </font><br> <font color=3D"#000000">>>>>>>>>>>> can tr= y </font><br> <font color=3D"#000000">>>>>>>>>>> to switch = to a config without bonding. </font><br> <font color=3D"#000000">>>>>>>>>>> </font=

<br> <font color=3D"#000000">>>>>>>>>>> "vm&q= uot; is a complicated term. </font><br> <font color=3D"#000000">>>>>>>>>>> </font= <br> <font color=3D"#000000">>>>>>>>>>> `brctl sho= w` would not show you a "vm" connected to a bridge. </font><b= r> <font color=3D"#000000">>>>> When </font><br> <font color=3D"#000000">>>>>>>>>>> you </= font><br> <font color=3D"#000000">>>>>>>>>>> WOULD see = is a vnet888 tap device. The "other side" of this </font><br> <font color=3D"#000000">>> device </font><br> <font color=3D"#000000">>>>> is </font><br> <font color=3D"#000000">>>>>>>>>>> held by qe= mu, which implement the VM. </font><br> <font color=3D"#000000">>>>>>>>>> Ok, understood= and found it, vnet2 </font><br> <font color=3D"#000000">>>>>>>>>> </font><br=

<font color=3D"#000000">>>>>>>>>>> I'm asking= if the dhcp offer has reached that tap device. </font><br> <font color=3D"#000000">>>>>>>>>> No, the DHCP o= ffer packet do not reach the vnet2 interface, I can </font><br> <font color=3D"#000000">>> see </font><br> <font color=3D"#000000">>>>>>>>>> only DHCP DISC= OVER. </font><br> <font color=3D"#000000">>>>>>>>> Ok, so it seems th= at we have a problem in the host bridging. </font><br> <font color=3D"#000000">>>>>>>>> </font><br> <font color=3D"#000000">>>>>>>>> Is it the latest k= ernel-3.10.0-229.7.2.el7.x86_64 ? </font><br> <font color=3D"#000000">>>>>>>>> </font><br> <font color=3D"#000000">>>>>>>>> Michael, a DHCP DI= SCOVER is sent out of a just-booted guest, and </font><br> <font color=3D"#000000">>>>> OFFER </font><br> <font color=3D"#000000">>>>>>>>> returns to the bri= dge, but is not propagated to the tap device. </font><br> <font color=3D"#000000">>>>>>>>> Can you suggest ho= w to debug this further? </font><br> <font color=3D"#000000">>>>>>>> Dump packets including= the ethernet headers. </font><br> <font color=3D"#000000">>>>>>>> Likely something inter= fered with them so the eth address is wrong. </font><br> <font color=3D"#000000">>>>>>>> </font><br> <font color=3D"#000000">>>>>>>> Since bonding does thi= s sometimes, this is the most likely culprit. </font><br> <font color=3D"#000000">>>>>>> We've ruled this out alrea= dy - Roberto reproduces the issue without a </font><br> <font color=3D"#000000">>>>>>> bond. </font><br> <font color=3D"#000000">>>>>> To me this looks like either a= regression in the host side bridging. But </font><br> <font color=3D"#000000">>> otoh it </font><br> <font color=3D"#000000">>>>> doesn't look </font><br> <font color=3D"#000000">>>>>> like it's happening always, be= cause otherwise I'd expect more noise </font><br> <font color=3D"#000000">>> around </font><br> <font color=3D"#000000">>>>> this issue. </font><br> <font color=3D"#000000">>>>>> - fabian </font><br> <font color=3D"#000000">>>>> Hard to say. E.g. forwarding delay= would do this for a while. </font><br> <font color=3D"#000000">>>>> If eth address of the packets is o= kay, poke at the fbd, maybe there's </font><br> <font color=3D"#000000">>>>> something wrong there. Maybe stp i= s detecting a loop - try checking that. </font><br> <font color=3D"#000000">>>> Someone is checking this ? </font>= <br> <font color=3D"#000000">>>> In tested config SPT was off. </fo= nt><br> <font color=3D"#000000">>> Then maybe you have a loop :) </font><= br> <font color=3D"#000000">> That was already checked, the MAC was unique i= n the VLAN. </font><br> <font color=3D"#000000">> </font><br> <font color=3D"#000000">> RN </font><br> <font color=3D"#000000">> </font><br> <br> Did you also try a reboot of the VM? We have the same issue with foreman
= 3;<br> and both Libvirt and oVirt. On second boot PXE boots properly from DHCP.
= 3;<br> <br> Haven't had the time to investigate yet so we're using mostly image <br=

based provisioning on oVirt at the moment. <br> <br> <br> = =3CBR /=3E =3CBR /=3E =3Cb style=3D=22color=3A=23604c78=22=3E=3C/b=3E=3Cbr=3E=3Cspan style=3D=22c= olor=3A=23604c78=3B=22=3E=3Cfont color=3D=22000000=22=3E=3Cspan style=3D=22= mso-fareast-language=3Aen-gb=3B=22 lang=3D=22NL=22=3EMet vriendelijke groet= =2C With kind regards=2C=3Cbr=3E=3Cbr=3E=3C/span=3EJorick Astrego=3C/font= =3E=3C/span=3E=3Cb style=3D=22color=3A=23604c78=22=3E=3Cbr=3E=3Cbr=3ENetbul= ae Virtualization Experts =3C/b=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3B= border-top=3A1px solid =23ccc=3B=22=3E=3Ctable style=3D=22width=3A 522px=22= =3E=3Ctbody=3E=3Ctr=3E=3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px= =22=3ETel=3A 053 20 30 270=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bf= ont-size=3A 10px=22=3Einfo=40netbulae=2Eeu=3C/td=3E =3Ctd style=3D=22wid= th=3A 130px=3Bfont-size=3A 10px=22=3EStaalsteden 4-3A=3C/td=3E =3Ctd sty= le=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EKvK 08198180=3C/td=3E=3C/tr= =3E=3Ctr=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EFax= =3A 053 20 30 271=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size= =3A 10px=22=3Ewww=2Enetbulae=2Eeu=3C/td=3E =3Ctd style=3D=22width=3A 130= px=3Bfont-size=3A 10px=22=3E7547 TA Enschede=3C/td=3E =3Ctd style=3D=22w= idth=3A 130px=3Bfont-size=3A 10px=22=3EBTW NL821234584B01=3C/td=3E=3C/tr=3E= =3C/tbody=3E=3C/table=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3Bborder-top= =3A1px solid =23ccc=3B=22=3E=3CBR /=3E =3C/body=3E =3C/html=3E ------------MIME-3653757893-462597306-delim--

-----Messaggio originale----- Da: Michael S. Tsirkin [mailto:mst@redhat.com] Inviato: mercoledì 29 luglio 2015 12:03 A: NUNIN Roberto Cc: Fabian Deutsch; users@ovirt.org Oggetto: Re: R: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer

On Wed, Jul 29, 2015 at 12:00:38PM +0200, NUNIN Roberto wrote:

...

...

-----Messaggio originale----- Da: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Per conto

di

...

...

Michael S. Tsirkin Inviato: giovedì 9 luglio 2015 15:15 A: Fabian Deutsch Cc: users@ovirt.org Oggetto: Re: [ovirt-users] R: R: R: R: R: R: PXE boot of a VM on vdsm don't read DHCP offer

...

----- Original Message -----

...

On Wed, Jul 08, 2015 at 09:11:42AM +0300, Michael S. Tsirkin wrote:

...

On Tue, Jul 07, 2015 at 05:13:28PM +0100, Dan Kenigsberg wrote: > On Tue, Jul 07, 2015 at 10:14:54AM +0200, NUNIN Roberto wrote: > > > > > > On Mon, Jul 06, 2015 at 10:33:59AM +0200, NUNIN Roberto wrote: > > > > Hi Dan > > > > > > > > Sorry for question: what do you mean for interface vnetxxxx ? > > > > Currently our path is : > > > > eno1 - eno2 ---- bond0 ----- bond.3500 (VLAN) ------ bridge ----- > > > > vm. > > > > > > > > Which one of these ? > > > > Moreover, reading Fabian statements about bonding limits, today I > > > > can try > > > to switch to a config without bonding. > > > > > > "vm" is a complicated term. > > > > > > `brctl show` would not show you a "vm" connected to a bridge. When > > > you > > > WOULD see is a vnet888 tap device. The "other side" of this device is > > > held by qemu, which implement the VM. > > > > Ok, understood and found it, vnet2 > > > > > > > > I'm asking if the dhcp offer has reached that tap device. > > > > No, the DHCP offer packet do not reach the vnet2 interface, I can see > > only DHCP DISCOVER. > > Ok, so it seems that we have a problem in the host bridging. > > Is it the latest kernel-3.10.0-229.7.2.el7.x86_64 ? > > Michael, a DHCP DISCOVER is sent out of a just-booted guest, and OFFER > returns to the bridge, but is not propagated to the tap device. > Can you suggest how to debug this further?

Dump packets including the ethernet headers. Likely something interfered with them so the eth address is wrong.

Since bonding does this sometimes, this is the most likely culprit.

We've ruled this out already - Roberto reproduces the issue without a bond.

To me this looks like either a regression in the host side bridging. But otoh it doesn't look like it's happening always, because otherwise I'd expect more noise around

On Thu, Jul 09, 2015 at 08:57:50AM -0400, Fabian Deutsch wrote: this issue.

...

- fabian

Hard to say. E.g. forwarding delay would do this for a while. If eth address of the packets is okay, poke at the fbd, maybe there's something wrong there. Maybe stp is detecting a loop - try checking that.

Someone is checking this ? In tested config SPT was off.

Then maybe you have a loop :)

No, it is not a loop. I've done further tests today and finally I've defined the following conditions. Erratic behavior is detected only within a cluster where nodes are HP Proliant BL660cGen8, connected to Cisco Nexus 7K thru HP FEX B22 blade interconnects and Cisco Nexus 5596 switches. All nic cards are 10Gbit. It doesn't happen with two HP Proliant DL380G5 with 10Gbit nics, connected directly to Cisco Nexus 5548UP switches and not happen with two HP Proliant ML350eGen8 nic 1Gbit connected to Cisco 4948 and next the same Nexus 5548UP. All nodes are running Centos 7.1 with latest updates and all networks are configured in the same mode, with bonding over two nic, then vlan interfaces and bridge towards VMs. Bonding is 4 for all and works correctly with DL380 and ML350 clusters. Well, I've tried to change the bonding mode on the BL660 cluster to mode 1 and the issue disappear. In all other bonding modes, it doesn't work; bridge interfaces receive DHCP offers and do NOT reject packets, but tap interfaces aren't receiving the offer. It works only with mode 1. How I can investigate further ? Desiderata is to have mode 4, to aggregate available bandwidth. RN

...

RN

...

-- MST _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

Thanks for answering, my considerations below. BR, Roberto

-----Messaggio originale----- Da: Fabian Deutsch [mailto:fdeutsch@redhat.com] Inviato: venerdì 3 luglio 2015 12:33 A: NUNIN Roberto Cc: Dan Kenigsberg; users@ovirt.org Oggetto: Re: [ovirt-users] R: R: R: PXE boot of a VM on vdsm don't read DHCP offer

----- Original Message -----

...

Hi Dan, guys

Sorry for very late follow-up, but we had a lot of other topics to fix just before to go back on this one.

We have tried another approach just to check if the kernel of the vdsm iso image used to install the host could create the problem I've reported to the list.

Now we have reinstalled the same hardware with latest CentOS 7.1, fully updated. Installed vdsm, then joined the oVirt cluster.

Well, we are observing the same behavior as before. No DHCP offer is reaching the booting VM, and:

brctl showmacs <bridge_if> show us the booting vm mac-address tcpdump -I <bridge_if> show us the dhcp offer coming from dhcp server.

We have also tried to remove ANY firewall rule.

It isn't a PXE issue (gPXE 0.9.7) but only a DHCP process issue. Infact, if we install a vm manually and assign a static IP, it works fine. If we switch to dhcp, the vm don't get the dynamic one. In this case, tcpdump on vm shows only the DHCP discovery, not the DHCP offer.

Any further suggestion/hint ?

I've observed this behavior in bug https://bugzilla.redhat.com/show_bug.cgi?id=1230638 We also removed all firewall rules, checked iPXE and I also saw the requests going out, but no replies getting to the VM. But here it sounds like it isn't specific to bonds. After all I did not find the solution yet.

In our config, I can see the DHCP offer until the hypervisor bridge interface toward vm

It is probably a good idea to install the oS with a static IP, and then switch to dhcp to then use tcpdump inside the vm to see what is reaching the inside.

Already done. Vm do not acquire the IP address and, on the vm side, tcpdump shows only requests. At the same time, the DHCP offer s detected on the bridge if of the hypervisor. With static IP, vm works fine.

- fabian

...

RN

...

-----Messaggio originale----- Da: Dan Kenigsberg [mailto:danken@redhat.com] Inviato: lunedì 18 maggio 2015 16:14 A: NUNIN Roberto Cc: users@ovirt.org; ibarkan@redhat.com Oggetto: Re: R: [ovirt-users] R: PXE boot of a VM on vdsm don't read

...

...

offer

On Fri, May 08, 2015 at 03:11:25PM +0200, NUNIN Roberto wrote:

...

Hi Dan Thanks for answering

...

Which kernel does the el7 host run? I think that Ido has seen a case where `brctl showmacs` was not populated with the VM mac, despite a packet coming out of it.

Kernel is: 3.10.0-123.20.1.el7.x86_64, package is vdsm only. Brctl isn't available within vdsm only package.

Could you try upgrading to a more up-to-date http://mirror.centos.org/centos- 7/7.1.1503/updates/x86_64/Packages/kernel-3.10.0- 229.4.2.el7.x86_64.rpm ?

bridge-utils is a vdsm dependency. It must exist on your host. Please see if the mac of the vNIC shows up on `brctl showmacs` as it should.

...

...

Can you tcpdump and check whether the bridge propogated the DHCP

offer

...

to the tap device of the said VM? Does the packet generated by `ether-wake MAC-of-VM` reach the tap device?

Yes: host "see" the broadcast : 0.000000 0.0.0.0 255.255.255.255 DHCP 346 DHCP Discover - Transaction ID 0x69267b67 It came from the right MAC: Source: Qumranet_15:81:03 (00:1a:4a:15:81:03) And it is tagged correctly: 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500

This is the offer, on the bond interface: 1.012355 10.155.124.2 10.155.124.246 DHCP 346 DHCP Offer - Transaction ID 0x69267b67 Layer 2 info: Ethernet II, Src: Cisco_56:83:c3 (84:78:ac:56:83:c3), Dst: Qumranet_15:81:03 (00:1a:4a:15:81:03) Tagging on the bond: 802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3500

The tag is correctly removed when DHCP offer is forwarded over the bond.3500. Here's the offer content, seems everything right:

Client IP address: 0.0.0.0 (0.0.0.0) Your (client) IP address: 10.155.124.246 (10.155.124.246) Next server IP address: 10.155.124.223 (10.155.124.223) Relay agent IP address: 10.155.124.2 (10.155.124.2) Client MAC address: Qumranet_15:81:03 (00:1a:4a:15:81:03) Client hardware address padding: 00000000000000000000 Server host name: 10.155.124.223 Boot file name: pxelinux.0 Magic cookie: DHCP

Nothing of this offer appear on the VM side.

But does it show on the host's bridge? on the tap device?

...

ether-wake -i bond0.3500 00:1a:4a:15:81:03 (started from the host) reach the VM eth0 interface: 2.002028 HewlettP_4a:47:b0 Qumranet_15:81:03 WOL 116

MagicPacket for Qumranet_15:81:03 (00:1a:4a:15:81:03)

...

Really strange behavior.

Roberto

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge.

This message is for the designated recipient only and may contain

DHCP privileged,

...

proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.

…

...

I've observed this behavior in bug https://bugzilla.redhat.com/show_bug.cgi?id=1230638 We also removed all firewall rules, checked iPXE and I also saw the requests going out, but no replies getting to the VM. But here it sounds like it isn't specific to bonds. After all I did not find the solution yet.

In our config, I can see the DHCP offer until the hypervisor bridge interface toward vm

Just to make sure, is your setup:

NIC -- bridge -- VM?

Or is a bond involved?

NICs (two) - BOND (type 4) - VLAN - BRIDGE - VM

...

It is probably a good idea to install the oS with a static IP, and then switch to dhcp to then use tcpdump inside the vm to see what is reaching the inside.

Already done. Vm do not acquire the IP address and, on the vm side, tcpdump shows only requests. At the same time, the DHCP offer s detected on the bridge if of the hypervisor.

With static IP, vm works fine.

Yes, it only seems to be around dhcp. If you don't have a bond involved, then please reopen the bug abaove, and we try to find someone to look at it. - fabian Questo messaggio e' indirizzato esclusivamente al destinatario indicato e potrebbe contenere informazioni confidenziali, riservate o proprietarie. Qualora la presente venisse ricevuta per errore, si prega di segnalarlo immediatamente al mittente, cancellando l'originale e ogni sua copia e distruggendo eventuali copie cartacee. Ogni altro uso e' strettamente proibito e potrebbe essere fonte di violazione di legge. This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately, deleting the original and all copies and destroying any hard copies. Any other use is strictly prohibited and may be unlawful.