Thanks Lucia, in the mean time I found another e-mail reporting the same credentials stored in that file (that are static and not randomly generated..).

Unfortunately the strategy works to have the hosted engine vm started, but it seems it is not recognized...

So I'm in the case of being during single host HCI deployment and in the final stage with the unsupported tsx-ctrl cpu flag, see this thread tto:

Strange that in /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-ansible-create_target_vm-202042810039-c8h9k9.log I have this

2020-05-28 10:01:16,234+0200 DEBUG var changed: host "localhost" var "server_cpu_dict" type "<class 'dict'>" value: "{
    "AMD EPYC": "EPYC",
    "AMD Opteron G4": "Opteron_G4",
    "AMD Opteron G5": "Opteron_G5",
    "IBM POWER8": "POWER8",
    "IBM POWER9": "POWER9",
    "IBM z114, z196": "z196-base",
    "IBM z13s, z13": "z13-base",
    "IBM z14": "z14-base",
    "IBM zBC12, zEC12": "zEC12-base",
    "Intel Broadwell Family": "Broadwell-noTSX",
    "Intel Cascadelake Server Family": "Cascadelake-Server,-hle,-rtm,+arch-capabilities",
    "Intel Haswell Family": "Haswell-noTSX",
    "Intel IvyBridge Family": "IvyBridge",
    "Intel Nehalem Family": "Nehalem",
    "Intel SandyBridge Family": "SandyBridge",
    "Intel Skylake Client Family": "Skylake-Client,-hle,-rtm",
    "Intel Skylake Server Family": "Skylake-Server,-hle,-rtm",
    "Intel Westmere Family": "Westmere",
    "Secure AMD EPYC": "EPYC,+ibpb,+virt-ssbd",
    "Secure Intel Broadwell Family": "Broadwell-noTSX,+spec-ctrl,+ssbd,+md-clear",
    "Secure Intel Cascadelake Server Family": "Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities",
    "Secure Intel Haswell Family": "Haswell-noTSX,+spec-ctrl,+ssbd,+md-clear",
    "Secure Intel IvyBridge Family": "IvyBridge,+pcid,+spec-ctrl,+ssbd,+md-clear",
    "Secure Intel Nehalem Family": "Nehalem,+spec-ctrl,+ssbd,+md-clear",
    "Secure Intel SandyBridge Family": "SandyBridge,+pcid,+spec-ctrl,+ssbd,+md-clear",
    "Secure Intel Skylake Client Family": "Skylake-Client,+spec-ctrl,+ssbd,+md-clear,-hle,-rtm",
    "Secure Intel Skylake Server Family": "Skylake-Server,+spec-ctrl,+ssbd,+md-clear,-hle,-rtm",
    "Secure Intel Westmere Family": "Westmere,+pcid,+spec-ctrl,+ssbd,+md-clear"
}"

and the tsx-ctrl flag is there.....

but when during deployment it starts the final engine vm after copying the local one to gluster storage, it gets:

May 28 10:06:59 novirt2 journal[13368]: unsupported configuration: unknown CPU feature: tsx-ctrl

The VM is started and then stopped and so on, so I try some times the dump command

virsh -r dumpxml HostedEngine > /tmp/he.xml

until I catch it

then I edit the file and remove the line with the tsx-ctrl flag and start the vm

[root@novirt2 log]# virsh create /tmp/he.xml
Please enter your authentication name: vdsm@ovirt
Please enter your password:
Domain HostedEngine created from /tmp/he.xml

So far so good the engine vm starts and it is reachable on its final hostname and engine service started and accessible web admin gui, but it is not recognized by the host

The qemu-kvm command is this one:

qemu     20826     1 99 10:16 ?        00:00:07 /usr/libexec/qemu-kvm -name guest=HostedEngine,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-HostedEngine/master-key.aes -machine pc-q35-rhel8.1.0,accel=kvm,usb=off,dump-guest-core=off -cpu Cascadelake-Server,md-clear=on,mds-no=on,hle=off,rtm=off,arch-capabilities=on -m size=16777216k,slots=16,maxmem=67108864k -overcommit mem-lock=off -smp 2,maxcpus=32,sockets=16,cores=2,threads=1 -object iothread,id=iothread1 -numa node,nodeid=0,cpus=0-31,mem=16384 -uuid b572d924-b278-41c7-a9da-52c4f590aac1 -smbios type=1,manufacturer=oVirt,product=RHEL,version=8-1.1911.0.9.el8,serial=d584e962-5461-4fa5-affa-db413e17590c,uuid=b572d924-b278-41c7-a9da-52c4f590aac1,family=oVirt -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,fd=40,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2020-05-28T08:16:29,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 -device pcie-root-port,port=0x18,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3 -device pcie-root-port,port=0x19,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 -device pcie-root-port,port=0x1a,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 -device pcie-root-port,port=0x1b,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 -device pcie-root-port,port=0x1c,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 -device pcie-root-port,port=0x1d,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 -device pcie-root-port,port=0x1e,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 -device pcie-root-port,port=0x1f,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 -device pcie-root-port,port=0x20,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 -device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 -device qemu-xhci,p2=8,p3=8,id=ua-b630a65c-8156-4542-b8e8-98b4d2c48f67,bus=pci.4,addr=0x0 -device virtio-scsi-pci,iothread=iothread1,id=ua-f6a70e5e-ebea-4468-a59a-cc760b9bcea4,bus=pci.5,addr=0x0 -device virtio-serial-pci,id=ua-608f9599-30b2-4ee6-a0d3-d5fb588583ad,max_ports=16,bus=pci.3,addr=0x0 -drive if=none,id=drive-ua-fa671f6c-dc42-4c59-a66d-ccfa3d5d422b,readonly=on -device ide-cd,bus=ide.2,drive=drive-ua-fa671f6c-dc42-4c59-a66d-ccfa3d5d422b,id=ua-fa671f6c-dc42-4c59-a66d-ccfa3d5d422b,werror=report,rerror=report -drive file=/var/run/vdsm/storage/3df8f6d4-d572-4d2b-9ab2-8abc456a396f/df02bff9-2c4b-4e14-a0a3-591a84ccaed9/bf435645-2999-4fb2-8d0e-5becab5cf389,format=raw,if=none,id=drive-ua-df02bff9-2c4b-4e14-a0a3-591a84ccaed9,cache=none,aio=threads -device virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.6,addr=0x0,drive=drive-ua-df02bff9-2c4b-4e14-a0a3-591a84ccaed9,id=ua-df02bff9-2c4b-4e14-a0a3-591a84ccaed9,bootindex=1,write-cache=on,serial=df02bff9-2c4b-4e14-a0a3-591a84ccaed9,werror=stop,rerror=stop -netdev tap,fds=42:43,id=hostua-b29ca99f-a53e-4de7-8655-b65ef4ba5dc4,vhost=on,vhostfds=44:45 -device virtio-net-pci,mq=on,vectors=6,host_mtu=1500,netdev=hostua-b29ca99f-a53e-4de7-8655-b65ef4ba5dc4,id=ua-b29ca99f-a53e-4de7-8655-b65ef4ba5dc4,mac=00:16:3e:0a:96:80,bus=pci.2,addr=0x0 -chardev socket,id=charserial0,fd=46,server,nowait -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=47,server,nowait -device virtserialport,bus=ua-608f9599-30b2-4ee6-a0d3-d5fb588583ad.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 -chardev socket,id=charchannel1,fd=48,server,nowait -device virtserialport,bus=ua-608f9599-30b2-4ee6-a0d3-d5fb588583ad.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=ua-608f9599-30b2-4ee6-a0d3-d5fb588583ad.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -chardev socket,id=charchannel3,fd=49,server,nowait -device virtserialport,bus=ua-608f9599-30b2-4ee6-a0d3-d5fb588583ad.0,nr=4,chardev=charchannel3,id=channel3,name=org.ovirt.hosted-engine-setup.0 -vnc 172.19.0.224:0 -k en-us -spice port=5901,tls-port=5902,addr=172.19.0.224,disable-ticketing,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=ua-ac7aed4d-d824-40f9-aaa8-1c0be702e38c,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 -device intel-hda,id=ua-c7078e28-5585-4866-bdbd-528ebddd8854,bus=pci.18,addr=0x1 -device hda-duplex,id=ua-c7078e28-5585-4866-bdbd-528ebddd8854-codec0,bus=ua-c7078e28-5585-4866-bdbd-528ebddd8854.0,cad=0 -device virtio-balloon-pci,id=ua-fc4f6b20-0b17-4198-b059-b5753893584d,bus=pci.7,addr=0x0 -object rng-random,id=objua-c4c3e5e7-1c19-4582-a87c-4f3fee4a0ee5,filename=/dev/urandom -device virtio-rng-pci,rng=objua-c4c3e5e7-1c19-4582-a87c-4f3fee4a0ee5,id=ua-c4c3e5e7-1c19-4582-a87c-4f3fee4a0ee5,bus=pci.8,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

The status remains as down

[root@novirt2 log]# hosted-engine --vm-status


--== Host novirt2.example.net (id: 1) status ==--

Host ID                            : 1
Host timestamp                     : 34180
Score                              : 0
Engine status                      : {"vm": "down_unexpected", "health": "bad", "detail": "Down", "reason": "bad vm status"}
Hostname                           : novirt2.example.net
Local maintenance                  : False
stopped                            : False
crc32                              : b297faaa
conf_on_shared_storage             : True
local_conf_timestamp               : 34180
Status up-to-date                  : True
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=34180 (Thu May 28 10:17:57 2020)
host-id=1
score=0
vm_conf_refresh_time=34180 (Thu May 28 10:17:57 2020)
conf_on_shared_storage=True
maintenance=False
state=EngineUnexpectedlyDown
stopped=False
timeout=Thu Jan  1 10:30:29 1970
[root@novirt2 log]#

Also restarting ovirt-ha-agent on host doesn't change the things....

In agent log i have

MainThread::ERROR::2020-05-28 10:23:45,817::hosted_engine::953::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Failed to stop engine VM: Command VM.destroy with args {'vmID': 'b572d924-b278-41c7-a9da-52c4f590aac1'} failed:
(code=1, message=Virtual machine does not exist: {'vmId': 'b572d924-b278-41c7-a9da-52c4f590aac1'})

MainThread::INFO::2020-05-28 10:23:45,843::brokerlink::73::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-ReinitializeFSM) sent? ignored
MainThread::INFO::2020-05-28 10:23:45,850::hosted_engine::517::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitoring_loop) Current state ReinitializeFSM (score: 0)

that is strange because the VM has started with that uuid. The qemu-kvm command infact contains the same uuid referred in agent.log: "uuid=b572d924-b278-41c7-a9da-52c4f590aac1"

In the mean time the deploy fails because from host point of view it was not able to reach up state..... and to test I put in global maintenance.

I can access the engine web admin portal but it seems it is recognized as a sort of external engine....?

Perhaps I have to remove some other line inside the generated he.xml file to have the engine recognized?

See screenshots from web admin view about hosted engine vm, the cluster type that has been setup by the deploy and the storage view (data and vm storage domains not appearing, only the engine one):

and content of he.xml

any input to solve and at least try some features of 4.4 on this hw env?

Thanks,

Gianluca