[ovirt-users] Networking issues with oVirt Guest VM.
Clint Smith
clint.smith.maui at gmail.com
Wed Jul 6 00:49:50 UTC 2016
Hi Yevgeny,
Thanks for your help.
No luck yet!
Q: IIUC, ovirt-engine and its host are vSphere VMs. Then, a kind of
no-macspoof should be applied from the vSphere side.
A: I'm not exactly sure what setting this would be on the vmware side but
we have ensured the following settings are enabled; Promiscuous Mode, MAC
Address Changes, Forged Transmits.
Q: BTW, are both of them on the same vShepre host?
A: Yes, they are both on the same vsphere host.
Q: Is DHCP server another VM on that host?
A: Yes, the DHCP server is another VM on that vsphere host as well.
Q: Where/how did you "turn on Port Mirroring"?
A: I turned on Port Mirroring in the vNIC Profile called ovirtmgmt, it's
off for most of my tests, however, when I turn it on, everything works.
Q: I'd start the troubleshooting by using tcpdump utility in order to
pinpoint the component that blocks the traffic.
A: I have done this and I can see all DHCP messages including the DHCPACK
on the bridge named 'ovirtmgmt', but when I do tcpdump on the guest VM
network device 'eth0' (which is also 'vnet0' on the host)I never see the
DHCPACK. This same thing happens with ARP replies from the gateway.
Here is an ACK message that is returned from the DHCP server to the
'ovirtmgmt' bridge (I don't know if this is helpful, but thought I'd
include it, just in case).
16:35:22.958616 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto
UDP (17), length 328)
10.255.233.125.67 > 10.255.233.185.68: [udp sum ok] BOOTP/DHCP, Reply,
length 300, xid 0x8d5c8e3d, Flags [none] (0x0000)
Your-IP 10.255.233.185
Client-Ethernet-Address 00:1a:4a:16:01:55
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: ACK
Server-ID Option 54, length 4: 10.255.233.125
Lease-Time Option 51, length 4: 600
Subnet-Mask Option 1, length 4: 255.255.255.0
BR Option 28, length 4: 10.255.233.255
Domain-Name-Server Option 6, length 4: 10.255.233.102
Default-Gateway Option 3, length 4: 10.255.233.1
Q: Did you try assigning a static IP instead of DHCP and then check
connectivity? If that works, then the problem is on the DHCP sever side
probably.
A: Yes, with a static IP on the guest, everything works fine. It's
possible, but in my naive understanding it seems unlikely, that the problem
is on the DHCP server side because the ACK makes it all the way back to the
bridge. Again, I could be wrong.
Q: If you do not see any requests in the DHCP server log, then I guess,
"dhclient -B" wouldn't help.
A: I see the whole DHCP process in my DHCP Server log:
Jul 5 21:21:12 mydomain dhcpd: DHCPDISCOVER from 00:1a:4a:16:01:55 via eth0
Jul 5 21:21:12 mydomain dhcpd: DHCPOFFER on 10.255.233.185 to
00:1a:4a:16:01:55 via eth0
Jul 5 21:21:12 mydomain dhcpd: DHCPREQUEST for 10.255.233.185
(10.255.233.125) from 00:1a:4a:16:01:55 via eth0
Jul 5 21:21:12 mydomain dhcpd: DHCPACK on 10.255.233.185 to
00:1a:4a:16:01:55 via eth0
In addition to "dhclient -B" allowing the guest to successfully gain an IP
address from the DHCP server, I recently found that running the command
"brctl ovirtmgmt setageing 0" works also. I read that this turns the
bridge into a hub, which is only a clue and not a solution, I think. It
seems that Forwarding Database changes that occur when I run that command
are what allows ACK message to then pass to the guest.
Here is my Forwarding Database when the Ageing is set to 300:
01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:02:02 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev bond0 self permanent
fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent
00:50:56:8e:aa:2a dev ens32 master ovirtmgmt
e4:d3:f1:d1:99:c4 dev ens32 master ovirtmgmt
00:50:56:8e:d6:a2 dev ens32 master ovirtmgmt
00:50:56:8e:d2:cd dev ens32 master ovirtmgmt
00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent
00:50:56:8e:17:59 dev ens32 master ovirtmgmt
fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent
00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent
00:50:56:8e:3d:f1 dev ens32 master ovirtmgmt
e4:d3:f1:d1:99:8c dev ens32 master ovirtmgmt
e4:d3:f1:d1:99:8b dev ens32 master ovirtmgmt
76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent
33:33:00:00:00:01 dev vnet0 self permanent
01:00:5e:00:00:01 dev vnet0 self permanent
33:33:ff:16:01:55 dev vnet0 self permanent
Here is the Forwarding Database(fdb) after setting the Ageing to 0:
01:00:5e:00:00:01 dev ens32 self permanent
33:33:00:00:02:02 dev ens32 self permanent
33:33:00:00:00:01 dev ens32 self permanent
33:33:00:00:00:01 dev bond0 self permanent
fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent
00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent
fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent
00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent
76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent
33:33:00:00:00:01 dev vnet0 self permanent
01:00:5e:00:00:01 dev vnet0 self permanent
33:33:ff:16:01:55 dev vnet0 self permanent
This is the result of 'brctl showmacs ovirtmgmt' with Ageing set to 300:
[root at ovirthost2 ~]# brctl showmacs ovirtmgmt
port no mac addr is local? ageing timer
1 00:50:56:8e:17:59 no 0.10
1 00:50:56:8e:2b:d7 no 22.30
1 00:50:56:8e:2e:ec no 3.66
1 00:50:56:8e:3d:f1 no 1.16
1 00:50:56:8e:a4:66 no 15.34
1 00:50:56:8e:a4:cc no 11.39
1 00:50:56:8e:aa:2a no 3.66
1 00:50:56:8e:be:ca yes 0.00
1 00:50:56:8e:be:ca yes 0.00
1 00:50:56:8e:d2:cd no 0.10
1 00:50:56:8e:d6:a2 no 3.70
1 e4:d3:f1:d1:99:8b no 0.26
1 e4:d3:f1:d1:99:8c no 0.26
1 e4:d3:f1:d1:99:c4 no 0.00
2 fe:1a:4a:16:01:55 yes 0.00
2 fe:1a:4a:16:01:55 yes 0.00
And here is the result of 'brctl showmacs ovirtmgmt' with Ageing set to 0:
[root at ovirthost2 ~]# brctl showmacs ovirtmgmt
port no mac addr is local? ageing timer
1 00:50:56:8e:be:ca yes 0.00
1 00:50:56:8e:be:ca yes 0.00
2 fe:1a:4a:16:01:55 yes 0.00
2 fe:1a:4a:16:01:55 yes 0.00
Maybe this will give more clues as to what's going on.
Q: Please turn iptables/firewalld off.
A: I have tried this ('service iptables stop') and it doesn't seem to make
a difference.
Thanks again for your help,
Clint
On Mon, Jul 4, 2016 at 2:41 AM, Yevgeny Zaspitsky <yzaspits at redhat.com>
wrote:
> Adding mailing list back...
>
> On Mon, Jul 4, 2016 at 3:38 PM, Yevgeny Zaspitsky <yzaspits at redhat.com>
> wrote:
>
>> Clint,
>>
>> Sorry, I missed that you already tried that.
>>
>> Here are my thoughts (some more shooting in the dark) after reading your
>> description again:
>>
>> - You have quite complicate setup. IIUC, ovirt-engine and its host
>> are vSphere VMs. Then, a kind of no-macspoof should be applied from the
>> vSphere side. BTW, are both of them on the same vShepre host? Is DHCP
>> server another VM on that host?
>> - Where/how did you "turn on Port Mirroring"?
>> - I'd start the troubleshooting by using tcpdump utility in order to
>> pinpoint the component that blocks the traffic.
>> - Did you try assigning a static IP instead of DHCP and then check
>> connectivity? If that works, then the problem is on the DHCP sever side
>> probably.
>> - If you do not see any requests in the DHCP server log, then I
>> guess, "dhclient -B" wouldn't help.
>> - Please turn iptables/firewalld off.
>>
>>
>> Regards,
>> Yevgeny
>>
>> On Sun, Jul 3, 2016 at 9:06 PM, Yevgeny Zaspitsky <yzaspits at redhat.com>
>> wrote:
>>
>>> Hello,
>>>
>>> IIUC using vdsm macspoof hook would help - reading [1] should help you
>>> configuring that.
>>>
>>> [1] https://github.com/oVirt/vdsm/blob/master/vdsm_hooks/macspoof/README
>>>
>>> Hope that helps,
>>> Yevgeny
>>>
>>> On Thu, Jun 30, 2016 at 6:11 AM, Clint Smith <clint.smith.maui at gmail.com
>>> > wrote:
>>>
>>>> Hello,
>>>>
>>>> I have been experimenting with oVirt for the last couple of weeks and I
>>>> must say it has a lot of nice features. I really like it, however, I am
>>>> having a heck of a time getting the guest networking all set up correctly.
>>>> I am hoping that someone can give me a little guidance in figuring this
>>>> out. I apologize in advance if some of my terminology is off, I am new.
>>>>
>>>> Here is a brief intro to my setup:
>>>> I created a Centos 7 VM within a vSphere/ESXI environment and then
>>>> installed ovirt-engine on it. I also created another Centos 7 VM and set
>>>> it up as a host. I have configured the Cluster and Host via the oVirt
>>>> Administration Portal. For simplicity, I am using the default ovirtmgmt
>>>> network as my only logical network, however I have tried several different
>>>> schemes with no luck. I have a DHCP server and a DNS server that are
>>>> siblings to the oVirt host and the engine. Both the engine and the host
>>>> have been upgraded to version 4.0.
>>>>
>>>> The problem:
>>>> My thought was that I would have the guest VMs on the oVirt host use my
>>>> existing DHCP server to get their IP addresses, at least at first. The
>>>> problem I am having is that the DHCPACK is not making it back across the
>>>> ovirtmgmt bridge and on to the guest. If I tell dhclient(from the guest)
>>>> to force a Broadcast (by using the –B option) on the DHCP server, it will
>>>> work. This is not a solution, just a clue. Another clue is that ARP
>>>> replies from the gateway don’t make it back to the machine, preventing
>>>> pings even when I force the IP. Lastly, If I turn on Port Mirroring,
>>>> everything works fine, but it’s my understanding that this is only for
>>>> debugging purposes.
>>>>
>>>> What I have tried (in no particular order):
>>>>
>>>> - Reading the docs
>>>> - Turning on VLAN tagging.
>>>> - Installing the mac-spoofing hook, making the configuration
>>>> changes to the engine, and then turning it on in the VM config. I also
>>>> verified that the ‘filterref’ tag was removed using virsh.
>>>> - Setting up a second logical network on a different subnet, and
>>>> connecting it to an additional network interface that I added to the host.
>>>> On the host, I setup dnsmasq as a DNS and DHCP server. I got this working
>>>> up to the point of having the same issues that I was having using the
>>>> existing DHCP and DNS servers on the ovirtmgmt network.
>>>> - I have tried various changes to iptables as well as the original
>>>> settings as well as verified that ebtables is not blocking any traffic. I
>>>> did configure iptables for logging and noticed it was dropping some traffic
>>>> related to DHCP, however it seemed like it was DISCOVER or REQUEST traffic
>>>> due to the IN, OUT, SRC, and DST variables in the log. I have viewed the
>>>> DHCP server logs multiple times and I can see that it is receiving the
>>>> DISCOVER and the REQUEST from my guests MAC and sending the OFFER and ACK
>>>> consistently.
>>>> - Setting SELinux to Permissive
>>>> - Setting ip_forward to 1
>>>> - Turning STP ON on the bridge
>>>> - Changing the bridge delay
>>>> - Setting up a dhcrelay using dnsmasq (not sure I implemented this
>>>> right though)
>>>>
>>>> I am really shooting in the dark when it comes to networking because I
>>>> am learning a lot of this on the fly. I feel like I must have a
>>>> misconception about how networking should work with oVirt. Is my entire
>>>> approach naïve? Any help/guidance that someone could offer would be much
>>>> appreciated.
>>>>
>>>> Thanks,
>>>> Clint
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160705/6500e713/attachment-0001.html>
More information about the Users
mailing list