[ovirt-users] Re: Can't run nested virtualization

On Fri, May 24, 2019 at 3:48 PM <nicolas(a)devels.es&gt; wrote: > El 2019-05-24 14:42, Simone Tiraboschi escribió: >> On Fri, May 24, 2019 at 3:39 PM <nicolas(a)devels.es&gt; wrote: >> >>> El 2019-05-24 14:30, Simone Tiraboschi escribió: >>>> On Fri, May 24, 2019 at 2:56 PM <nicolas(a)devels.es&gt; wrote: >>>> >>>>> El 2019-05-24 13:39, Simone Tiraboschi escribió: >>>>>> On Fri, May 24, 2019 at 2:32 PM <nicolas(a)devels.es&gt; wrote: >>>>>> >>>>>>> El 2019-05-24 13:22, Simone Tiraboschi escribió: >>>>>>>> On Fri, May 24, 2019 at 1:51 PM <nicolas(a)devels.es&gt; wrote: >>>>>>>> >>>>>>>>> El 2019-05-24 12:41, nicolas(a)devels.es escribió: >>>>>>>>>> El 2019-05-14 08:19, Yedidyah Bar David escribió: >>>>>>>>>>> On Tue, May 14, 2019 at 10:02 AM <nicolas(a)devels.es&gt; >>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Please, any ideas about this? >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> El 2019-05-10 09:18, nicolas(a)devels.es escribió: >>>>>>>>>>>> > Hi, >>>>>>>>>>>> > >>>>>>>>>>>> > We're running oVirt version 4.3.3, and trying to >>>>> configure >>>>>>>>> one of the >>>>>>>>>>>> > hosts to support Nested Virtualization, but when >>>>> installing >>>>>>>>> the nested >>>>>>>>>>>> > host it claims it doesn't support hardware >>>>> virtualization. >>>>>>>>>>>> > >>>>>>>>>>>> > On the physical host, we've enabled nested >>>>> virtualization: >>>>>>>>>>>> > >>>>>>>>>>>> > # cat /sys/module/kvm_intel/parameters/nested >>>>>>>>>>>> > Y >>>>>>>>>>>> > >>>>>>>>>>>> > Content of /etc/modprobe.d/kvm.conf: >>>>>>>>>>>> > >>>>>>>>>>>> > options kvm_intel nested=1 >>>>>>>>>>>> > options kvm_intel enable_shadow_vmcs=1 >>>>>>>>>>>> > options kvm_intel enable_apicv=1 >>>>>>>>>>>> > options kvm_intel ept=1 >>>>>>>>>>>> > >>>>>>>>>>>> > I created a VM to run on that host, which will be the >>>>>>> nested >>>>>>>>> host. I >>>>>>>>>>>> > try to deploy it but the engine will show it failed >>>>>>> because: >>>>>>>>>>>> > >>>>>>>>>>>> > 2019-05-10 09:11:32,006+01 ERROR >>>>>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>>>>> > (VdsDeploy) [6381e662] EVENT_ID: >>>>>>>>> VDS_INSTALL_IN_PROGRESS_ERROR(511), >>>>>>>>>>>> > An error has occurred during installation of Host >>> host1: >>>>>>>>> Failed to >>>>>>>>>>>> > execute stage 'Setup validation': Hardware does not >>>>> support >>>>>>>>>>>> > virtualization. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Yedidyah, sorry for the delayed answer. >>>>>>>>>> >>>>>>>>>>> You might find some more details in the host-deploy log, >>>>>>>>>>> which you should be able to find in >>>>>>>>> /var/log/ovirt-engine/host-deploy >>>>>>>>>>> (on the engine machine, it's copied there after deploy >>>>>>>>> finishes). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I had a look at it, but nothing relevant shows up besides >>>>> this >>>>>>>>> line: >>>>>>>>>> >>>>>>>>>> 2019-05-10 09:11:32,628+0100 DEBUG otopi.context >>>>>>>>>> context._executeMethod:145 method exception >>>>>>>>>> Traceback (most recent call last): >>>>>>>>>>    File >>> "/tmp/ovirt-qPjYkVy6Ys/pythonlib/otopi/context.py", >>>>>>> line >>>>>>>>> 132, >>>>>>>>>> in _executeMethod >>>>>>>>>>      method['method']() >>>>>>>>>>    File >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > "/tmp/ovirt-qPjYkVy6Ys/otopi-plugins/ovirt-host-deploy/vdsm/hardware.py", >>>>>>>>>> line 71, in _validate_virtualization >>>>>>>>>>      _('Hardware does not support virtualization') >>>>>>>>>> RuntimeError: Hardware does not support virtualization >>>>>>>>>> >>>>>>>>>>> It's been some time since I configured this myself, so I >>> do >>>>>>> not >>>>>>>>>>> remember >>>>>>>>>>> the details anymore. Did you check some guides/blog >>>>> posts/etc. >>>>>>>>> about >>>>>>>>>>> this? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I didn't. I just enabled nested virtualization in the > host >>>>> and >>>>>>>>> tried to >>>>>>>>>> deploy. >>>>>>>>>> >>>>>>>>>>> What type of CPU did you configure in the VM (and >>> cluster)? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> In the Cluster I have the Intel Broadwell Family, and as >>> the >>>>> VM >>>>>>>>> CPU I >>>>>>>>>> have the default cluster CPU which is the one I just >>>>>>> referenced. >>>>>>>>> Not >>>>>>>>>> sure if anything else should be done. >>>>>>>>>> >>>>>>>>>>> To see what checks the code does, you can read [1], >>> although >>>>>>> the >>>>>>>>> log >>>>>>>>>>> should be enough IMO. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>> >>> /usr/lib/python2.7/site-packages/ovirt_host_deploy/hardware.py >>>>>>>>> >>>>>>>>> I just saw this in the log: >>>>>>>>> >>>>>>>>> 2019-05-24 12:44:56,000+0100 DEBUG >>>>>>> otopi.ovirt_host_deploy.hardware >>>>>>>>> >>>>>>>>> hardware._isVirtualizationEnabled:208 virtualization > support >>>>>>>>> GenuineIntel (cpu: False, bios: True) >>>>>>>>> >>>>>>>>> So the CPU is not able to support virtualization. Why > could >>> be >>>>>>>>> that? >>>>>>>> >>>>>>>> Hi, >>>>>>>> can you please share the output of cat /proc/cpuinfo ? >>>>>>>>   >>>>>>> >>>>>>> Hi Simone, >>>>>>> >>>>>>> Please find attached the output (assuming you meant the VM >>> that >>>>> is >>>>>>> meant >>>>>>> to be added as a host). You'll probably see the model name > as >>>>>>> Haswell >>>>>>> because I forced it to test, but the same happens with >>>>> Broadwell. >>>>>> >>>>>> OK, >>>>>> vmx flag is not there so this VM doesn't support > virtualization >>>>> and >>>>>> for that you have to enable nested virt support on the >>> underlying >>>>>> level. What (SW) are you using on the underlying layer? >>>>>> >>>>>> Let's try to recap: >>>>>> L0 is your physical machine >>>>>> L1 is the VM you are going to use as a virtualization host > for >>>>> oVirt >>>>>> L2 are the VM you are going to create with oVirt >>>>>> >>>>>> In order to be able to start a L2 VM from oVirt, you have to >>>>> enable >>>>>> nested virtualization support on your L0 physical host. >>>>>> >>>>> >>>>> Ok, the output I sent in the previous e-mail was for L1. For > L0 >>> I'm >>>>> >>>>> sending the cpuinfo log now. >>>>> >>>>> I saw in the code, the flag that is returning false at >>> deployment >>>>> of L1 >>>>> is this: >>>>> >>>>>      cpu_ok = self._cpu_has_vmx_support() >>>>> >>>>> Which calls the _cpu_has_vmx_support(): >>>>> >>>>>      def _cpu_has_vmx_support(self): >>>>>          eax, ebx, ecx, edx = self._cpuid(1) >>>>>          # CPUID.1:ECX.VMX[bit 5] -> VT >>>>>          ret = ecx & (1 << 5) != 0 >>>>>          self.logger.debug('vmx support: %s', ret) >>>>>          return ret >>>>> >>>>>  From that, am I assuming well that vmx support should be >>> enabled >>>>> also >>>>> for L1? If so, what could be the reason for not being enabled? >>>> >>>> Nested virtualization support on L0 is supposed to enable >>>> virtualization support (vmx flag) on L1. >>>> How you defined and started your L1 VMs? >>>>   >>> >>> I basically created the L1 VM when L0 had no nested support yet, > I >>> enabled the nested support once the L1 machine was already > created, >>> can >>> that make a difference? >> >> I suppose so. >> But which kind of software are you using on your L0? directly > libvirt >> with virsh? virt-install? virt-manager? another oVirt isntance? >> > > L0 is a physical host for oVirt 4.1.9 (cannot upgrade yet) with > vdsm. I > can try to destroy L1 and reinstall it now that L0 has nested > virtualization support and see how it goes, that will probably be > on > next monday, I'll send the result once I try. Did you installed vdsm-hook-nestedvt on your L0 hosts? if vdsm-hook-nestedvt is there and cat /sys/module/kvm_intel/parameters/nested reports Y, shutting down and  restarting your L1 VMs from oVirt should be enough.