Ovirt 4.4 - starting just-installed host from ovirt console fails
Good afternoon all, I recently had to rebuild my cluster due to a self inflicted error. I have finally managed to get the ovirt host software installed and communicating on all hosts. The first host installed and started cleanly. However, after installation the second host is failing to start. Prior to my cluster crash, this host was running well in the cluster. During the downtime, we applied microcode and BIOS updates as part of the recovery process. I have reviewed this chain: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/N3PPT34GBRLPLTS... and reached a dead end. Based on what I see (in the long stream of logs and info following), it looks like I should change the cluster CPU type from Cascadelake to Haswell to restore normal operation. The long involved stuff: The Engine reports: [image: image.png] Host CPU type is not compatible with Cluster Properties. [image: image.png] The host CPU does not match the Cluster CPU Type and is running in a degraded mode. It is missing the following CPU flags: model_Cascadelake-Server-noTSX. Please update the host CPU microcode or change the Cluster CPU Type. The Cluster definition is: [image: image.png] *lscpu returns:* Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz BIOS Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz Stepping: 2 CPU MHz: 3300.000 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 4988.45 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 30720K NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d *cpuid returns:* CPU 0: vendor_id = "GenuineIntel" version information (1/eax): processor type = primary processor (0) family = 0x6 (6) model = 0xf (15) stepping id = 0x2 (2) extended family = 0x0 (0) extended model = 0x3 (3) (family synth) = 0x6 (6) (model synth) = 0x3f (63) (simple synth) = Intel (unknown type) (Haswell C1/M1/R2) {Haswell}, 22nm *virsh domcapabilities returns:* <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>kvm</domain> <machine>pc-i440fx-rhel7.6.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <iothreads supported='yes'/> <os supported='yes'> <enum name='firmware'/> <loader supported='yes'> <value>/usr/share/OVMF/OVMF_CODE.secboot.fd</value> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> <enum name='secure'> <value>no</value> </enum> </loader> </os> <cpu> <mode name='host-passthrough' supported='yes'> <enum name='hostPassthroughMigratable'> <value>on</value> <value>off</value> </enum> </mode> <mode name='maximum' supported='yes'> <enum name='maximumMigratable'> <value>on</value> <value>off</value> </enum> </mode> <mode name='host-model' supported='yes'> <model fallback='forbid'>Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='ibpb'/> <feature policy='require' name='ibrs'/> <feature policy='require' name='amd-stibp'/> <feature policy='require' name='amd-ssbd'/> <feature policy='require' name='skip-l1dfl-vmentry'/> <feature policy='require' name='pschange-mc-no'/> </mode> <mode name='custom' supported='yes'> <model usable='yes'>qemu64</model> <model usable='yes'>qemu32</model> <model usable='no'>phenom</model> <model usable='yes'>pentium3</model> <model usable='yes'>pentium2</model> <model usable='yes'>pentium</model> <model usable='yes'>n270</model> <model usable='yes'>kvm64</model> <model usable='yes'>kvm32</model> <model usable='yes'>coreduo</model> <model usable='yes'>core2duo</model> <model usable='no'>athlon</model> <model usable='yes'>Westmere-IBRS</model> <model usable='yes'>Westmere</model> <model usable='no'>Snowridge</model> <model usable='no'>Skylake-Server-noTSX-IBRS</model> <model usable='no'>Skylake-Server-IBRS</model> <model usable='no'>Skylake-Server</model> <model usable='no'>Skylake-Client-noTSX-IBRS</model> <model usable='no'>Skylake-Client-IBRS</model> <model usable='no'>Skylake-Client</model> <model usable='yes'>SandyBridge-IBRS</model> <model usable='yes'>SandyBridge</model> <model usable='yes'>Penryn</model> <model usable='no'>Opteron_G5</model> <model usable='no'>Opteron_G4</model> <model usable='no'>Opteron_G3</model> <model usable='yes'>Opteron_G2</model> <model usable='yes'>Opteron_G1</model> <model usable='yes'>Nehalem-IBRS</model> <model usable='yes'>Nehalem</model> <model usable='yes'>IvyBridge-IBRS</model> <model usable='yes'>IvyBridge</model> <model usable='no'>Icelake-Server-noTSX</model> <model usable='no'>Icelake-Server</model> <model usable='no' deprecated='yes'>Icelake-Client-noTSX</model> <model usable='no' deprecated='yes'>Icelake-Client</model> <model usable='yes'>Haswell-noTSX-IBRS</model> <model usable='yes'>Haswell-noTSX</model> <model usable='no'>Haswell-IBRS</model> <model usable='no'>Haswell</model> <model usable='no'>EPYC-Rome</model> <model usable='no'>EPYC-Milan</model> <model usable='no'>EPYC-IBPB</model> <model usable='no'>EPYC</model> <model usable='no'>Dhyana</model> <model usable='no'>Cooperlake</model> <model usable='yes'>Conroe</model> <model usable='no'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model> <model usable='no'>Broadwell-noTSX-IBRS</model> <model usable='no'>Broadwell-noTSX</model> <model usable='no'>Broadwell-IBRS</model> <model usable='no'>Broadwell</model> <model usable='yes'>486</model> </mode> </cpu> <memoryBacking supported='yes'> <enum name='sourceType'> <value>file</value> <value>anonymous</value> <value>memfd</value> </enum> </memoryBacking> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> <value>sata</value> </enum> <enum name='model'> <value>virtio</value> <value>virtio-transitional</value> <value>virtio-non-transitional</value> </enum> </disk> <graphics supported='yes'> <enum name='type'> <value>vnc</value> <value>spice</value> <value>egl-headless</value> </enum> </graphics> <video supported='yes'> <enum name='modelType'> <value>vga</value> <value>cirrus</value> <value>qxl</value> <value>virtio</value> <value>none</value> <value>bochs</value> <value>ramfb</value> </enum> </video> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'/> </hostdev> <rng supported='yes'> <enum name='model'> <value>virtio</value> <value>virtio-transitional</value> <value>virtio-non-transitional</value> </enum> <enum name='backendModel'> <value>random</value> <value>egd</value> <value>builtin</value> </enum> </rng> <filesystem supported='yes'> <enum name='driverType'> <value>path</value> <value>handle</value> <value>virtiofs</value> </enum> </filesystem> <tpm supported='yes'> <enum name='model'> <value>tpm-tis</value> <value>tpm-crb</value> </enum> <enum name='backendModel'> <value>passthrough</value> <value>emulator</value> </enum> </tpm> </devices> <features> <gic supported='no'/> <vmcoreinfo supported='yes'/> <genid supported='yes'/> <backingStoreInput supported='yes'/> <backup supported='yes'/> <sev supported='no'/> </features> </domainCapabilities> Please advise.
Good news: Yes, setting the cluster compatibility to Haswell has mostly restored functionality. The last issue I need to address is a hardware issue - the storage network is physically connected to the wrong adapter. Ain't no software in the world gonna jump the high impedance air gap. Thank you all for your help and patience. On Tue, Jun 21, 2022 at 1:24 PM David Johnson <djohnson@maxistechnology.com> wrote:
Good afternoon all,
I recently had to rebuild my cluster due to a self inflicted error.
I have finally managed to get the ovirt host software installed and communicating on all hosts.
The first host installed and started cleanly. However, after installation the second host is failing to start. Prior to my cluster crash, this host was running well in the cluster.
During the downtime, we applied microcode and BIOS updates as part of the recovery process.
I have reviewed this chain: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/N3PPT34GBRLPLTS... and reached a dead end.
Based on what I see (in the long stream of logs and info following), it looks like I should change the cluster CPU type from Cascadelake to Haswell to restore normal operation.
The long involved stuff:
The Engine reports:
[image: image.png] Host CPU type is not compatible with Cluster Properties. [image: image.png] The host CPU does not match the Cluster CPU Type and is running in a degraded mode. It is missing the following CPU flags: model_Cascadelake-Server-noTSX. Please update the host CPU microcode or change the Cluster CPU Type.
The Cluster definition is: [image: image.png]
*lscpu returns:* Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz BIOS Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz Stepping: 2 CPU MHz: 3300.000 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 4988.45 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 30720K NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
*cpuid returns:* CPU 0: vendor_id = "GenuineIntel" version information (1/eax): processor type = primary processor (0) family = 0x6 (6) model = 0xf (15) stepping id = 0x2 (2) extended family = 0x0 (0) extended model = 0x3 (3) (family synth) = 0x6 (6) (model synth) = 0x3f (63) (simple synth) = Intel (unknown type) (Haswell C1/M1/R2) {Haswell}, 22nm
*virsh domcapabilities returns:*
<domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>kvm</domain> <machine>pc-i440fx-rhel7.6.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <iothreads supported='yes'/> <os supported='yes'> <enum name='firmware'/> <loader supported='yes'> <value>/usr/share/OVMF/OVMF_CODE.secboot.fd</value> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> <enum name='secure'> <value>no</value> </enum> </loader> </os> <cpu> <mode name='host-passthrough' supported='yes'> <enum name='hostPassthroughMigratable'> <value>on</value> <value>off</value> </enum> </mode> <mode name='maximum' supported='yes'> <enum name='maximumMigratable'> <value>on</value> <value>off</value> </enum> </mode> <mode name='host-model' supported='yes'> <model fallback='forbid'>Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='ibpb'/> <feature policy='require' name='ibrs'/> <feature policy='require' name='amd-stibp'/> <feature policy='require' name='amd-ssbd'/> <feature policy='require' name='skip-l1dfl-vmentry'/> <feature policy='require' name='pschange-mc-no'/> </mode> <mode name='custom' supported='yes'> <model usable='yes'>qemu64</model> <model usable='yes'>qemu32</model> <model usable='no'>phenom</model> <model usable='yes'>pentium3</model> <model usable='yes'>pentium2</model> <model usable='yes'>pentium</model> <model usable='yes'>n270</model> <model usable='yes'>kvm64</model> <model usable='yes'>kvm32</model> <model usable='yes'>coreduo</model> <model usable='yes'>core2duo</model> <model usable='no'>athlon</model> <model usable='yes'>Westmere-IBRS</model> <model usable='yes'>Westmere</model> <model usable='no'>Snowridge</model> <model usable='no'>Skylake-Server-noTSX-IBRS</model> <model usable='no'>Skylake-Server-IBRS</model> <model usable='no'>Skylake-Server</model> <model usable='no'>Skylake-Client-noTSX-IBRS</model> <model usable='no'>Skylake-Client-IBRS</model> <model usable='no'>Skylake-Client</model> <model usable='yes'>SandyBridge-IBRS</model> <model usable='yes'>SandyBridge</model> <model usable='yes'>Penryn</model> <model usable='no'>Opteron_G5</model> <model usable='no'>Opteron_G4</model> <model usable='no'>Opteron_G3</model> <model usable='yes'>Opteron_G2</model> <model usable='yes'>Opteron_G1</model> <model usable='yes'>Nehalem-IBRS</model> <model usable='yes'>Nehalem</model> <model usable='yes'>IvyBridge-IBRS</model> <model usable='yes'>IvyBridge</model> <model usable='no'>Icelake-Server-noTSX</model> <model usable='no'>Icelake-Server</model> <model usable='no' deprecated='yes'>Icelake-Client-noTSX</model> <model usable='no' deprecated='yes'>Icelake-Client</model> <model usable='yes'>Haswell-noTSX-IBRS</model> <model usable='yes'>Haswell-noTSX</model> <model usable='no'>Haswell-IBRS</model> <model usable='no'>Haswell</model> <model usable='no'>EPYC-Rome</model> <model usable='no'>EPYC-Milan</model> <model usable='no'>EPYC-IBPB</model> <model usable='no'>EPYC</model> <model usable='no'>Dhyana</model> <model usable='no'>Cooperlake</model> <model usable='yes'>Conroe</model> <model usable='no'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model> <model usable='no'>Broadwell-noTSX-IBRS</model> <model usable='no'>Broadwell-noTSX</model> <model usable='no'>Broadwell-IBRS</model> <model usable='no'>Broadwell</model> <model usable='yes'>486</model> </mode> </cpu> <memoryBacking supported='yes'> <enum name='sourceType'> <value>file</value> <value>anonymous</value> <value>memfd</value> </enum> </memoryBacking> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> <value>sata</value> </enum> <enum name='model'> <value>virtio</value> <value>virtio-transitional</value> <value>virtio-non-transitional</value> </enum> </disk> <graphics supported='yes'> <enum name='type'> <value>vnc</value> <value>spice</value> <value>egl-headless</value> </enum> </graphics> <video supported='yes'> <enum name='modelType'> <value>vga</value> <value>cirrus</value> <value>qxl</value> <value>virtio</value> <value>none</value> <value>bochs</value> <value>ramfb</value> </enum> </video> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'/> </hostdev> <rng supported='yes'> <enum name='model'> <value>virtio</value> <value>virtio-transitional</value> <value>virtio-non-transitional</value> </enum> <enum name='backendModel'> <value>random</value> <value>egd</value> <value>builtin</value> </enum> </rng> <filesystem supported='yes'> <enum name='driverType'> <value>path</value> <value>handle</value> <value>virtiofs</value> </enum> </filesystem> <tpm supported='yes'> <enum name='model'> <value>tpm-tis</value> <value>tpm-crb</value> </enum> <enum name='backendModel'> <value>passthrough</value> <value>emulator</value> </enum> </tpm> </devices> <features> <gic supported='no'/> <vmcoreinfo supported='yes'/> <genid supported='yes'/> <backingStoreInput supported='yes'/> <backup supported='yes'/> <sev supported='no'/> </features> </domainCapabilities>
Please advise.
participants (1)
-
David Johnson