[ovirt-users] Re: HostedEngine migration fails with VM destroyed during the startup.

Good day Strahil, I believe I found the causing link: HostedEngine.log-20200216:-cpu SandyBridge,pcid=on,spec-ctrl=on,ssbd=on,md-clear=on,vme=on,hypervisor=on,arat=on,xsaveopt=on \ HostedEngine.log-20200216:2020-02-13T17:58:38.674630Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.md-clear [bit 10] HostedEngine.log-20200216:2020-02-13T17:58:38.676205Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.md-clear [bit 10] HostedEngine.log-20200216:2020-02-13T17:58:38.676901Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.md-clear [bit 10] HostedEngine.log-20200216:2020-02-13T17:58:38.677616Z qemu-kvm: warning: host doesn't support requested feature: CPUID.07H:EDX.md-clear [bit 10] The "md-clear" CPU seem to be removed as feature due to spectre vulnerabilities. However, when I check the CPU Type/flags of the VMs on the same Host as where Engine is currently, as well as on the other hosts, the md-clear seems to be only present on the HostedEngine: * HostedEngine: FromwebUI: Intel SandyBridge IBRS SSBD Family Via virsh: #virsh dumpxml <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>SandyBridge</model> <topology sockets='16' cores='4' threads='1'/> <feature policy='require' name='pcid'/> <feature policy='require' name='spec-ctrl'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='vme'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='xsaveopt'/> <numa> <cell id='0' cpus='0-3' memory='16777216' unit='KiB'/> </numa> </cpu> * OtherVMs: From webUI: (SandyBridge,+pcid,+spec-ctrl,+ssbd) Via virsh: #virsh dumpxml <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>SandyBridge</model> <topology sockets='16' cores='1' threads='1'/> <feature policy='require' name='pcid'/> <feature policy='require' name='spec-ctrl'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='vme'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='xsaveopt'/> <numa> <cell id='0' cpus='0-3' memory='4194304' unit='KiB'/> </numa> </cpu> Strahil, knowing this, do you propose different approach or shall I just proceed with initially suggested workaround? Kindly awaiting your eply. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 e: m.vrgotic@activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. On 16/02/2020, 15:28, "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; wrote: ssh root@engine "poweroff" ssh host-that-holded-engine "virsh undefine HostedEngine; virsh list --all" Lot's of virsh - less vdsm :) Good luck Best Regards, Strahil Nikolov В неделя, 16 февруари 2020 г., 16:01:44 ч. Гринуич+2, Vrgotic, Marko <m.vrgotic(a)activevideo.com&gt; написа: Hi Strahil, Regarding step 3: Stop and undefine the VM on the last working host One question: How do I undefine HostedEngine from last Host? Hosted-engine command does not provide such option, or it's just not obvious. Kindly awaiting your reply. ----- kind regards/met vriendelijke groeten Marko Vrgotic ActiveVideo On 14/02/2020, 18:44, "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; wrote: On February 14, 2020 4:19:53 PM GMT+02:00, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; wrote: I am afraid that your suspicions are right. What is the host cpu and the HostedEngine's xml? Have you checked the xml on any working VM ? What cpu flags do the working VMs have ? How to solve - I think I have a solution , but you might not like it. 1. Get current VM xml with virsh 2. Set all nodes in maintenance 'hosted-engine --set-maintenance --mode=global' 3. Stop and undefine the VM on the last working host 4. Edit the xml from step 1 and add/remove the flags that are different from the other (working) VMs 5. Define the HostedEngine on any of the updated hosts 6. Start the HostedEngine via virsh. 7. Try with different cpu flags until the engine starts. 8. Leave the engine for at least 12 hours , so it will have enough time to update it's own configuration. 9. Remove the maintenance and migrate the engine to the other upgraded host 10. Patch the last HostedEngine's host I have done this procedure in order to recover my engine (except changing the cpu flags). Note: You may hit some hiccups: A) virsh alias alias virsh='virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf' B) HostedEngine network missing: [root@ovirt1 ~]# virsh net-dumpxml vdsm-ovirtmgmt <network> <name>vdsm-ovirtmgmt</name> <uuid>986c27cf-a1ec-44d8-ae61-ee09ce75c886</uuid> <forward mode='bridge'/> <bridge name='ovirtmgmt'/> </network> Define in xml and add it via: virsh net-define somefile.xml C) Missing disk Vdsm is creating symlinks like these: [root@ovirt1 808423f9-8a5c-40cd-bc9f-2568c85b8c74]# pwd /var/run/vdsm/storage/808423f9-8a5c-40cd-bc9f-2568c85b8c74 [root@ovirt1 808423f9-8a5c-40cd-bc9f-2568c85b8c74]# ls -l total 20 lrwxrwxrwx. 1 vdsm kvm 129 Feb 2 19:05 2c74697a-8bd9-4472-8a98-bf624f3462d5 -> /rhev/data-center/mnt/glusterSD/gluster1:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/2c74697a-8bd9-4472-8a98-bf624f3462d5 lrwxrwxrwx. 1 vdsm kvm 129 Feb 2 19:09 3ec27d6d-921c-4348-b799-f50543b6f919 -> /rhev/data-center/mnt/glusterSD/gluster1:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/3ec27d6d-921c-4348-b799-f50543b6f919 lrwxrwxrwx. 1 vdsm kvm 129 Feb 2 19:09 441abdc8-6cb1-49a4-903f-a1ec0ed88429 -> /rhev/data-center/mnt/glusterSD/gluster1:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/441abdc8-6cb1-49a4-903f-a1ec0ed88429 lrwxrwxrwx. 1 vdsm kvm 129 Feb 2 19:09 94ade632-6ecc-4901-8cec-8e39f3d69cb0 -> /rhev/data-center/mnt/glusterSD/gluster1:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/94ade632-6ecc-4901-8cec-8e39f3d69cb0 lrwxrwxrwx. 1 vdsm kvm 129 Feb 2 19:05 fe62a281-51e9-4b23-87b3-2deb52357304 -> /rhev/data-center/mnt/glusterSD/gluster1:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/fe62a281-51e9-4b23-87b3-2deb52357304 [root@ovirt1 808423f9-8a5c-40cd-bc9f-2568c85b8c74]# Just create the link, so it points to correct destinationand power up again. Good luck ! Best Regards, Strahil Nikolov