<div dir="ltr">Just bumping this in case anyone has any ideas as to what I might be able to do to potentially get this to work.</div><div class="gmail_extra"><br><div class="gmail_quote">On 12 May 2017 at 13:57, Mark Duggan <span dir="ltr"><<a href="mailto:mduggan@gmail.com" target="_blank">mduggan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Gianluca,<div><br></div><div>So I installed the engine into a separate VM, and didn't go down the hosted-engine path, although if I was to look at this with physical hosts, this seems like a really good approach.</div><div><br></div><div>To answer Michal's question from earlier, the nested VM inside the oVirt Hypervisors has been up for 23+ hours and it has not progressed past the Bios. </div><div>Also, with respect to the vdsm-hooks, here's a list. </div><div><br></div><div>Dumpxml attached (hopefully with identifying information removed)</div><div><br></div><div><div>vdsm-hook-nestedvt.noarch</div><div>vdsm-hook-vmfex-dev.noarch</div><div>vdsm-hook-allocate_net.noarch</div><div>vdsm-hook-checkimages.noarch</div><div>vdsm-hook-checkips.x86_64</div><div>vdsm-hook-diskunmap.noarch</div><div>vdsm-hook-ethtool-options.<wbr>noarch</div><div>vdsm-hook-extnet.noarch</div><div>vdsm-hook-extra-ipv4-addrs.<wbr>x86_64</div><div>vdsm-hook-fakesriov.x86_64</div><div>vdsm-hook-fakevmstats.noarch</div><div>vdsm-hook-faqemu.noarch</div><div>vdsm-hook-fcoe.noarch</div><div>vdsm-hook-fileinject.noarch</div><div>vdsm-hook-floppy.noarch</div><div>vdsm-hook-hostusb.noarch</div><div>vdsm-hook-httpsisoboot.noarch</div><div>vdsm-hook-hugepages.noarch</div><div>vdsm-hook-ipv6.noarch</div><div>vdsm-hook-isolatedprivatevlan.<wbr>noarch</div><div>vdsm-hook-localdisk.noarch</div><div>vdsm-hook-macbind.noarch</div><div>vdsm-hook-macspoof.noarch</div><div>vdsm-hook-noipspoof.noarch</div><div>vdsm-hook-numa.noarch</div><div>vdsm-hook-openstacknet.noarch</div><div>vdsm-hook-pincpu.noarch</div><div>vdsm-hook-promisc.noarch</div><div>vdsm-hook-qemucmdline.noarch</div><div>vdsm-hook-qos.noarch</div><div>vdsm-hook-scratchpad.noarch</div><div>vdsm-hook-smbios.noarch</div><div>vdsm-hook-spiceoptions.noarch</div><div>vdsm-hook-vhostmd.noarch</div><div>vdsm-hook-vmdisk.noarch</div><div>vdsm-hook-vmfex.noarch</div></div><div><br></div><div>I'm running ESXi 5.5. For the hypervisor VMs I've set the "Expose Hardware Assisted Virtualization to the guest OS"</div><div><br></div><div>Hypervisor VMs are running CentOS 7.3</div><div><br></div><div><img src="cid:ii_15bfdcb13f1be3d2" alt="Inline images 1" width="562" height="351"></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 12 May 2017 at 09:36, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, May 12, 2017 at 1:06 PM, Michal Skrivanek <span dir="ltr"><<a href="mailto:michal.skrivanek@redhat.com" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-"><br>
> On 11 May 2017, at 19:52, Mark Duggan <<a href="mailto:mduggan@gmail.com" target="_blank">mduggan@gmail.com</a>> wrote:<br>
><br>
> Hi everyone,<br>
><br>
> From reading through the mailing list, it does appear that it's possible to have the ovirt nodes/hosts be VMware virtual machines, once I enable the appropriate settings on the VMware side. All seems to have gone well, I can see the hosts in the ovirt interface, but when I attempt to create and start a VM it never gets past printing the SeaBios version and the machine UUID to the screen/console. It doesn't appear to try to boot from the hard disk or an ISO that I've attached.<br>
><br>
> Has anyone else encountered similar behaviour?<br>
<br>
</span>I wouldn’t think you can even get that far.<br>
It may work with full emulation (non-kvm) but we kind of enforce it in oVirt so some changes are likely needed.<br>
Of course even if you succeed it’s going to be hopelessly slow. (or maybe it is indeed working and just runs very slow)<br>
<br>
Nested on a KVM hypervisor runs ok<br>
<br>
Thanks,<br>
michal<br><br></blockquote><div><br></div></span><div>In the past I was able to get an Openstack Icehouse environment running inside vSphere 5.x for a POC (on poweful physical servers) and performance of nested VMs inside the virtual compute nodes was acceptable.</div><div>More recently I configured a standalone ESXi server 6.0 U2 on a Nuc6 with 32Gb of ram and 2 ssd disks and on it I have now running 2 kinds of environments (just verified they are still on after some months I abandoned them to their destiny... ;-)</div><div><br></div><div>1) an ESXi VM acting as a single oVirt host (4.1.1 final or pre, I don't remember) with self hosted engine (that itself becomes an L2 VM) and also another VM (CentOS 6.8)</div><div>See here a screenshot of the web admin gui with a spice console open after connecting to the engine:</div><div><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvanpTUnFuZ2FURms/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvanpTUnFuZ2FURms<wbr>/view?usp=sharing</a><br></div><div><br></div><div>2) a virtual oVirt gluster environment based on 4.0.5 with 3 Virtual Hosts (with one as arbiter node if I remember correctly) </div></div><br></div><div class="gmail_extra">On this second environment I have ovirt01, virt02 and ovirt03 VMs:</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">[root@ovirt02 ~]# hosted-engine --vm-status</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">--== Host 1 status ==--</div><div class="gmail_extra"><br></div><div class="gmail_extra">Status up-to-date : True</div><div class="gmail_extra">Hostname : ovirt01.localdomain.local</div><div class="gmail_extra">Host ID : 1</div><div class="gmail_extra">Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}</div><div class="gmail_extra">Score : 3042</div><div class="gmail_extra">stopped : False</div><div class="gmail_extra">Local maintenance : False</div><div class="gmail_extra">crc32 : 2041d7b6</div><div class="gmail_extra">Host timestamp : 15340856</div><div class="gmail_extra">Extra metadata (valid at timestamp):</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metadata_parse_version=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metadata_feature_version=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>timestamp=15340856 (Fri May 12 14:59:17 2017)</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>host-id=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>score=3042</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>maintenance=False</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>state=EngineDown</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>stopped=False</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">--== Host 2 status ==--</div><div class="gmail_extra"><br></div><div class="gmail_extra">Status up-to-date : True</div><div class="gmail_extra">Hostname : 192.168.150.103</div><div class="gmail_extra">Host ID : 2</div><div class="gmail_extra">Engine status : {"health": "good", "vm": "up", "detail": "up"}</div><div class="gmail_extra">Score : 3400</div><div class="gmail_extra">stopped : False</div><div class="gmail_extra">Local maintenance : False</div><div class="gmail_extra">crc32 : 27a80001</div><div class="gmail_extra">Host timestamp : 15340760</div><div class="gmail_extra">Extra metadata (valid at timestamp):</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metadata_parse_version=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metadata_feature_version=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>timestamp=15340760 (Fri May 12 14:59:11 2017)</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>host-id=2</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>score=3400</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>maintenance=False</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>state=EngineUp</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>stopped=False</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">--== Host 3 status ==--</div><div class="gmail_extra"><br></div><div class="gmail_extra">Status up-to-date : True</div><div class="gmail_extra">Hostname : ovirt03.localdomain.local</div><div class="gmail_extra">Host ID : 3</div><div class="gmail_extra">Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}</div><div class="gmail_extra">Score : 2986</div><div class="gmail_extra">stopped : False</div><div class="gmail_extra">Local maintenance : False</div><div class="gmail_extra">crc32 : 98aed4ec</div><div class="gmail_extra">Host timestamp : 15340475</div><div class="gmail_extra">Extra metadata (valid at timestamp):</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metadata_parse_version=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metadata_feature_version=1</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>timestamp=15340475 (Fri May 12 14:59:22 2017)</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>host-id=3</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>score=2986</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>maintenance=False</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>state=EngineDown</div><div class="gmail_extra"><span class="m_-486339675139254758m_-4810947598826804692gmail-m_-2049154232866614922gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>stopped=False</div><div class="gmail_extra">[root@ovirt02 ~]# </div><div><br></div><div>The virtual node ovirt02 has the hosted engine vm running on it</div><div>It was some months I didn't come back, but it seems it is still up... ;-)</div><div><br></div><div><br></div><div><div>[root@ovirt02 ~]# uptime</div><div> 15:02:18 up 177 days, 13:26, 1 user, load average: 2.04, 1.46, 1.22</div><div><br></div><div>[root@ovirt02 ~]# free</div><div> total used free shared buff/cache available</div><div>Mem: 12288324 6941068 3977644 595204 1369612 4340808</div><div>Swap: 5242876 2980672 2262204</div><div>[root@ovirt02 ~]# </div></div><div><br></div><div><div>[root@ovirt02 ~]# ps -ef|grep qemu-kvm</div><div>qemu 18982 1 8 2016 ? 14-20:33:44 /usr/libexec/qemu-kvm -name HostedEngine -S -machine pc-i440fx-rhel7.2.0,accel=kvm,<wbr>usb=off </div></div><div><br></div><div>the first node (used for deploy with hostname ovirt01 and with name inside oVirt web admin gui of hosted_engine_1) has other 3 L2 VMs running</div><div><div>[root@ovirt01 ~]# ps -ef|grep qemu-kvm</div><div>qemu 125069 1 1 15:01 ? 00:00:11 /usr/libexec/qemu-kvm -name atomic2 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,<wbr>usb=off </div><div>qemu 125186 1 2 15:02 ? 00:00:18 /usr/libexec/qemu-kvm -name centos6 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,<wbr>usb=off </div><div>qemu 125329 1 1 15:02 ? 00:00:06 /usr/libexec/qemu-kvm -name cirros3 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,<wbr>usb=off </div></div><div><br></div><div>I also tested live migration with success.</div><div><br></div><div>Furthermore all the 3 ESXI VMs hat are the 3 oVirt Hypervisors have still in place a VMware snapshot, because I was making a test with the idea of reverting after preliminary testing and this adds further load... </div><div>see here some screenshots:</div><div><br></div><div>ESXi with its 3 VMs that are the 3 oVirt hypervisors</div><div><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvWEtwM3otLU5uUkU/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvWEtwM3otLU5uUkU<wbr>/view?usp=sharing</a><br></div><div><br></div><div>oVirt Engine web admin portal with one L2 VM console open</div><div><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvS2I1eEREclBqSU0/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvS2I1eEREclBqSU0<wbr>/view?usp=sharing</a><br></div><div><br></div><div>oVirt Engine web admin Hosts tab</div><div><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvWGcxV0xDUGpINlU/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvWGcxV0xDUGpINlU<wbr>/view?usp=sharing</a><br></div><div><br></div><div>oVrt Engine Gluster data domain</div><div><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvVkxMa1R2eGRfV2s/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvVkxMa1R2eGRfV2s<wbr>/view?usp=sharing</a><br></div><div><br></div><div><br></div><div>Let me see and find the configuration settings I set up for it, because some months have gone and I then had little time to follow it...</div><div><br></div><div>In the mean time, what is the version of your ESXi environment? Because settings to put in place changed form version 5 to version 6.</div><div>What are particular settings you already configured for the ESXi VMs you plan to use as oVirt hypervisors?</div><span class="m_-486339675139254758HOEnZb"><font color="#888888"><div> </div><div>Gianluca</div></font></span></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>