ESXi 6.7 as a nested VM on top of oVirt cluster.

Hello list I am wondering if anyone has tried this before? I am trying to consolidate my lab to an oVirt cluster which consists of oVirt (4.3.8) and 2 KVM hypervisors (CentOS 7.7.). Among other things, one of my efforts is a small ESXi + vCenter lab. In addition, I have a standalone KVM hypervisor. I can run nested ESXi 6.7 on top of the standalone KVM hypervisor (with nested KVM enabled) without any problem. However, on top of oVirt-controlled KVMs I have some issues. The hypervisors have nested kvm support enabled via vdsm hooks. qemu emulates e1000 for a nested ESXi vm. ESXi installation process goes smooth but as soon as I enable management network and restart it, the nested ESXi cannot communicate with the outside world (DNS ping cannot pass). Also https connection to VMWare ESXi web gui and ping to the ESXi also fail. I noticed on my client machine, that ARP requests are incomplete for ESXI as well. Within oVirt, I see no packet drops in "Network interface" tab for the nested ESXi. In addition, I have a few, ordinary, non-nested VMs running on the same network as the nested vm and I can normally establish connection to those machines. Additionally, as a test, I created a nested KVM vm on top of the oVirt cluster, same network as the nested ESXi and it works as expected - can spin VMs, can connect to it. The network assigned to the nested ESXi has a "No network filter" vNic profile applied. I tried to google for a solution but found only this: https://github.com/mechinn/kvmhidden - not sure if this is the solution at all (also, I wonder if this is still up-to-date after 3+ years)? Could I please ask if anyone tried something similar and experienced this problem? Is there any additional configuration that I should apply to the oVirt cluster? Thank you very much in advance! Regards, Branimir

On Mon, Feb 3, 2020 at 3:16 PM <branimirp@gmail.com> wrote:
Hello list
I am wondering if anyone has tried this before? I am trying to consolidate my lab to an oVirt cluster which consists of oVirt (4.3.8) and 2 KVM hypervisors (CentOS 7.7.). Among other things, one of my efforts is a small ESXi + vCenter lab. In addition, I have a standalone KVM hypervisor. I can run nested ESXi 6.7 on top of the standalone KVM hypervisor (with nested KVM enabled) without any problem. However, on top of oVirt-controlled KVMs I have some issues. The hypervisors have nested kvm support enabled via vdsm hooks. qemu emulates e1000 for a nested ESXi vm. ESXi installation process goes smooth but as soon as I enable management network and restart it, the nested ESXi cannot communicate with the outside world (DNS ping cannot pass). Also https connection to VMWare ESXi web gui and ping to the ESXi also fail. I noticed on my client machine, that ARP requests are incomplete for ESXI as well. Within oVirt, I see no packet drops in "Network interface" tab for the nested ESXi.
Can you compare the qemu-kvm command line between: - the nested ESXi 6.7 on top of the standalone KVM hypervisor - the nested ESXi 6.7 on top of oVirt based hypervisor Also, which OS version of the standalone KVM hypervisor? which cpu for the KVM hypervisor which oVirt version? which cpu for the oVirt hypervisor? did you use e1000 as the network interface type also in oVirt for the ESXi 6.7 VM? Gianluca

Hi Gianluca Thank you very much for your reply!
On Mon, Feb 3, 2020 at 3:16 PM <branimirp(a)gmail.com> wrote:
Can you compare the qemu-kvm command line between:
The command lines are (proverbially) huge but I spotted a few (notable) differences in '-cpu' option - if we ignore qemu cmd (please see my comment on standalone KVM OS):
- the nested ESXi 6.7 on top of the standalone KVM hypervisor
/usr/bin/qemu-system-x86_64 -cpu Haswell-noTSX-IBRS,vme=on,ss=on,vmx=on,rdrand=on,hypervisor=on,arat=on,tsc_adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,fma=off,aes=off,avx=off,f16c=off,bmi1=off,avx2=off,smep=off,bmi2=off
- the nested ESXi 6.7 on top of oVirt based hypervisor
Not sure if "host" is a "shortcut" for all CPU flags exposed to the nested VM: /usr/libexec/qemu-kvm -cpu host,vmx=on (in oVirt management, "Pass-Through Host CPU" is selected, In Edit Host, "Kernel" tab, Kernel command line has "kvm-intel.nested=1" and as mentioned vdsm nestedvt hook rpm was installed)
Also, which OS version of the standalone KVM hypervisor?
openSUSE 15.1
which cpu for the KVM hypervisor Intel(R) Celeron(R) CPU G1820 @ 2.70GHz
which oVirt version? the newest oVirt 4.3.8 with both CentOS 7.7 hypervisors fully updated.
which cpu for the oVirt hypervisor? Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
did you use e1000 as the network interface type also in oVirt for the ESXi 6.7 VM?
Correct. For ESXi, I use e1000 on both standalone KVM and oVirt KVM hypervisor.
If you need any additional info, please let me know. Thank you! Regards, Branimir
participants (2)
-
branimirp@gmail.com
-
Gianluca Cecchi