<div dir="ltr"><div>Hi Yevgeny,</div><div>Thanks for your help.</div><div><br></div><div>No luck yet!</div><div><br></div><div>Q: IIUC, ovirt-engine and its host are vSphere VMs. Then, a kind of no-macspoof should be applied from the vSphere side.</div><div>A: I'm not exactly sure what setting this would be on the vmware side but we have ensured the following settings are enabled; Promiscuous Mode, MAC Address Changes, Forged Transmits.</div><div><br></div><div>Q: BTW, are both of them on the same vShepre host?</div><div>A: Yes, they are both on the same vsphere host.</div><div><br></div><div>Q: Is DHCP server another VM on that host?</div><div>A: Yes, the DHCP server is another VM on that vsphere host as well.</div><div><br></div><div>Q: Where/how did you "turn on Port Mirroring"?</div><div>A: I turned on Port Mirroring in the vNIC Profile called ovirtmgmt, it's off for most of my tests, however, when I turn it on, everything works.</div><div><br></div><div>Q: I'd start the troubleshooting by using tcpdump utility in order to pinpoint the component that blocks the traffic.</div><div>A: I have done this and I can see all DHCP messages including the DHCPACK on the bridge named 'ovirtmgmt', but when I do tcpdump on the guest VM network device 'eth0' (which is also 'vnet0' on the host)I never see the DHCPACK. This same thing happens with ARP replies from the gateway.</div><div><br></div><div>Here is an ACK message that is returned from the DHCP server to the 'ovirtmgmt' bridge (I don't know if this is helpful, but thought I'd include it, just in case).</div><div>16:35:22.958616 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)</div><div> 10.255.233.125.67 > 10.255.233.185.68: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x8d5c8e3d, Flags [none] (0x0000)</div><div> Your-IP 10.255.233.185</div><div> Client-Ethernet-Address 00:1a:4a:16:01:55</div><div> Vendor-rfc1048 Extensions</div><div> Magic Cookie 0x63825363</div><div> DHCP-Message Option 53, length 1: ACK</div><div> Server-ID Option 54, length 4: 10.255.233.125</div><div> Lease-Time Option 51, length 4: 600</div><div> Subnet-Mask Option 1, length 4: 255.255.255.0</div><div> BR Option 28, length 4: 10.255.233.255</div><div> Domain-Name-Server Option 6, length 4: 10.255.233.102</div><div> Default-Gateway Option 3, length 4: 10.255.233.1</div><div><br></div><div><br></div><div>Q: Did you try assigning a static IP instead of DHCP and then check connectivity? If that works, then the problem is on the DHCP sever side probably.</div><div>A: Yes, with a static IP on the guest, everything works fine. It's possible, but in my naive understanding it seems unlikely, that the problem is on the DHCP server side because the ACK makes it all the way back to the bridge. Again, I could be wrong.</div><div><br></div><div><br></div><div>Q: If you do not see any requests in the DHCP server log, then I guess, "dhclient -B" wouldn't help.</div><div>A: I see the whole DHCP process in my DHCP Server log:</div><div><br></div><div>Jul 5 21:21:12 mydomain dhcpd: DHCPDISCOVER from 00:1a:4a:16:01:55 via eth0</div><div>Jul 5 21:21:12 mydomain dhcpd: DHCPOFFER on 10.255.233.185 to 00:1a:4a:16:01:55 via eth0</div><div>Jul 5 21:21:12 mydomain dhcpd: DHCPREQUEST for 10.255.233.185 (10.255.233.125) from 00:1a:4a:16:01:55 via eth0</div><div>Jul 5 21:21:12 mydomain dhcpd: DHCPACK on 10.255.233.185 to 00:1a:4a:16:01:55 via eth0</div><div><br></div><div>In addition to "dhclient -B" allowing the guest to successfully gain an IP address from the DHCP server, I recently found that running the command "brctl ovirtmgmt setageing 0" works also. I read that this turns the bridge into a hub, which is only a clue and not a solution, I think. It seems that Forwarding Database changes that occur when I run that command are what allows ACK message to then pass to the guest.</div><div><br></div><div>Here is my Forwarding Database when the Ageing is set to 300:</div><div><br></div><div>01:00:5e:00:00:01 dev ens32 self permanent</div><div>33:33:00:00:02:02 dev ens32 self permanent</div><div>33:33:00:00:00:01 dev ens32 self permanent</div><div>33:33:00:00:00:01 dev bond0 self permanent</div><div>fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent</div><div>00:50:56:8e:aa:2a dev ens32 master ovirtmgmt </div><div>e4:d3:f1:d1:99:c4 dev ens32 master ovirtmgmt </div><div>00:50:56:8e:d6:a2 dev ens32 master ovirtmgmt </div><div>00:50:56:8e:d2:cd dev ens32 master ovirtmgmt </div><div>00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent</div><div>00:50:56:8e:17:59 dev ens32 master ovirtmgmt </div><div>fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent</div><div>fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent</div><div>00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent</div><div>00:50:56:8e:3d:f1 dev ens32 master ovirtmgmt </div><div>e4:d3:f1:d1:99:8c dev ens32 master ovirtmgmt </div><div>e4:d3:f1:d1:99:8b dev ens32 master ovirtmgmt </div><div>76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent</div><div>33:33:00:00:00:01 dev vnet0 self permanent</div><div>01:00:5e:00:00:01 dev vnet0 self permanent</div><div>33:33:ff:16:01:55 dev vnet0 self permanent</div><div><br></div><div><br></div><div>Here is the Forwarding Database(fdb) after setting the Ageing to 0:</div><div><br></div><div>01:00:5e:00:00:01 dev ens32 self permanent</div><div>33:33:00:00:02:02 dev ens32 self permanent</div><div>33:33:00:00:00:01 dev ens32 self permanent</div><div>33:33:00:00:00:01 dev bond0 self permanent</div><div>fe:4e:66:bf:cf:48 dev ;vdsmdummy; vlan 1 master ;vdsmdummy; permanent</div><div>00:50:56:8e:be:ca dev ens32 master ovirtmgmt permanent</div><div>fe:1a:4a:16:01:55 dev vnet0 master ovirtmgmt permanent</div><div>fe:1a:4a:16:01:55 dev vnet0 vlan 1 master ovirtmgmt permanent</div><div>00:50:56:8e:be:ca dev ens32 vlan 1 master ovirtmgmt permanent</div><div>76:f6:65:58:fe:f5 dev ovirtmgmt vlan 1 master ovirtmgmt permanent</div><div>33:33:00:00:00:01 dev vnet0 self permanent</div><div>01:00:5e:00:00:01 dev vnet0 self permanent</div><div>33:33:ff:16:01:55 dev vnet0 self permanent</div><div><br></div><div>This is the result of 'brctl showmacs ovirtmgmt' with Ageing set to 300:</div><div><br></div><div>[root@ovirthost2 ~]# brctl showmacs ovirtmgmt</div><div>port no<span class="" style="white-space:pre"> </span>mac addr<span class="" style="white-space:pre"> </span>is local?<span class="" style="white-space:pre"> </span>ageing timer</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:17:59<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 0.10</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:2b:d7<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 22.30</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:2e:ec<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 3.66</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:3d:f1<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 1.16</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:a4:66<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 15.34</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:a4:cc<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 11.39</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:aa:2a<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 3.66</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:be:ca<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:be:ca<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:d2:cd<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 0.10</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:d6:a2<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 3.70</div><div> 1<span class="" style="white-space:pre"> </span>e4:d3:f1:d1:99:8b<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 0.26</div><div> 1<span class="" style="white-space:pre"> </span>e4:d3:f1:d1:99:8c<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 0.26</div><div> 1<span class="" style="white-space:pre"> </span>e4:d3:f1:d1:99:c4<span class="" style="white-space:pre"> </span>no<span class="" style="white-space:pre"> </span> 0.00</div><div> 2<span class="" style="white-space:pre"> </span>fe:1a:4a:16:01:55<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div> 2<span class="" style="white-space:pre"> </span>fe:1a:4a:16:01:55<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div><br></div><div>And here is the result of 'brctl showmacs ovirtmgmt' with Ageing set to 0:</div><div><br></div><div>[root@ovirthost2 ~]# brctl showmacs ovirtmgmt</div><div>port no<span class="" style="white-space:pre"> </span>mac addr<span class="" style="white-space:pre"> </span>is local?<span class="" style="white-space:pre"> </span>ageing timer</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:be:ca<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div> 1<span class="" style="white-space:pre"> </span>00:50:56:8e:be:ca<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div> 2<span class="" style="white-space:pre"> </span>fe:1a:4a:16:01:55<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div> 2<span class="" style="white-space:pre"> </span>fe:1a:4a:16:01:55<span class="" style="white-space:pre"> </span>yes<span class="" style="white-space:pre"> </span> 0.00</div><div><br></div><div>Maybe this will give more clues as to what's going on.</div><div><br></div><div><br></div><div>Q: Please turn iptables/firewalld off.</div><div>A: I have tried this ('service iptables stop') and it doesn't seem to make a difference.</div><div><br></div><div><br></div><div>Thanks again for your help,</div><div>Clint</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 4, 2016 at 2:41 AM, Yevgeny Zaspitsky <span dir="ltr"><<a href="mailto:yzaspits@redhat.com" target="_blank">yzaspits@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Adding mailing list back...<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 4, 2016 at 3:38 PM, Yevgeny Zaspitsky <span dir="ltr"><<a href="mailto:yzaspits@redhat.com" target="_blank">yzaspits@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Clint,<br><br></div>Sorry, I missed that you already tried that.<br></div><br></div>Here are my thoughts (some more shooting in the dark) after reading your description again:<br><ul><li>You have quite complicate setup. IIUC, ovirt-engine and its host are vSphere VMs. Then, a kind of no-macspoof should be applied from the vSphere side. BTW, are both of them on the same vShepre host? Is DHCP server another VM on that host?<br></li><li>Where/how did you "turn on Port Mirroring"?<br></li><li>I'd start the troubleshooting by using tcpdump utility in order to pinpoint the component that blocks the traffic.</li><li>Did you try assigning a static IP instead of DHCP and then check connectivity? If that works, then the problem is on the DHCP sever side probably.</li><li>If you do not see any requests in the DHCP server log, then I guess, "dhclient -B" wouldn't help.</li><li>Please turn iptables/firewalld off.</li></ul></div><div><br></div><div>Regards,<br></div><div>Yevgeny<br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 3, 2016 at 9:06 PM, Yevgeny Zaspitsky <span dir="ltr"><<a href="mailto:yzaspits@redhat.com" target="_blank">yzaspits@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hello,<br><br></div>IIUC using vdsm macspoof hook would help - reading [1] should help you configuring that.<br><br>[1] <a href="https://github.com/oVirt/vdsm/blob/master/vdsm_hooks/macspoof/README" target="_blank">https://github.com/oVirt/vdsm/blob/master/vdsm_hooks/macspoof/README</a><br><br></div>Hope that helps,<br></div>Yevgeny<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Jun 30, 2016 at 6:11 AM, Clint Smith <span dir="ltr"><<a href="mailto:clint.smith.maui@gmail.com" target="_blank">clint.smith.maui@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><div>Hello,</div><div><br></div><div>I have been experimenting with oVirt for the last couple of weeks and I must say it has a lot of nice features. I really like it, however, I am having a heck of a time getting the guest networking all set up correctly. I am hoping that someone can give me a little guidance in figuring this out. I apologize in advance if some of my terminology is off, I am new.</div><div><br></div><div>Here is a brief intro to my setup:</div><div>I created a Centos 7 VM within a vSphere/ESXI environment and then installed ovirt-engine on it. I also created another Centos 7 VM and set it up as a host. I have configured the Cluster and Host via the oVirt Administration Portal. For simplicity, I am using the default ovirtmgmt network as my only logical network, however I have tried several different schemes with no luck. I have a DHCP server and a DNS server that are siblings to the oVirt host and the engine. Both the engine and the host have been upgraded to version 4.0.</div><div><br></div><div>The problem:</div><div>My thought was that I would have the guest VMs on the oVirt host use my existing DHCP server to get their IP addresses, at least at first. The problem I am having is that the DHCPACK is not making it back across the ovirtmgmt bridge and on to the guest. If I tell dhclient(from the guest) to force a Broadcast (by using the –B option) on the DHCP server, it will work. This is not a solution, just a clue. Another clue is that ARP replies from the gateway don’t make it back to the machine, preventing pings even when I force the IP. Lastly, If I turn on Port Mirroring, everything works fine, but it’s my understanding that this is only for debugging purposes.</div><div><br></div><div>What I have tried (in no particular order):</div><ul><li>Reading the docs</li><li>Turning on VLAN tagging.</li><li>Installing the mac-spoofing hook, making the configuration changes to the engine, and then turning it on in the VM config. I also verified that the ‘filterref’ tag was removed using virsh.</li><li>Setting up a second logical network on a different subnet, and connecting it to an additional network interface that I added to the host. On the host, I setup dnsmasq as a DNS and DHCP server. I got this working up to the point of having the same issues that I was having using the existing DHCP and DNS servers on the ovirtmgmt network.</li><li>I have tried various changes to iptables as well as the original settings as well as verified that ebtables is not blocking any traffic. I did configure iptables for logging and noticed it was dropping some traffic related to DHCP, however it seemed like it was DISCOVER or REQUEST traffic due to the IN, OUT, SRC, and DST variables in the log. I have viewed the DHCP server logs multiple times and I can see that it is receiving the DISCOVER and the REQUEST from my guests MAC and sending the OFFER and ACK consistently.</li><li>Setting SELinux to Permissive</li><li>Setting ip_forward to 1</li><li>Turning STP ON on the bridge</li><li>Changing the bridge delay</li><li>Setting up a dhcrelay using dnsmasq (not sure I implemented this right though)</li></ul><div>I am really shooting in the dark when it comes to networking because I am learning a lot of this on the fly. I feel like I must have a misconception about how networking should work with oVirt. Is my entire approach naïve? Any help/guidance that someone could offer would be much appreciated.</div><div><br></div><div>Thanks,</div><div>Clint</div><div><br></div><div><br></div><div><br></div><div><div></div></div></div>
<br></div></div>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>