Hi Thank for your answer. the virsh command ask me to authenticate, i tried my ldap account without success. There is an internal account? For the error message no capabibilities is missing, it said bad cpu type but did not change it. "Host CPU type is not compatible with Cluster Properties." Lionel De: "Lucia Jelinkova" <ljelinko@redhat.com> À: "Lionel Caignec" <caignec@cines.fr> Cc: "users" <users@ovirt.org> Envoyé: Lundi 14 Décembre 2020 11:30:19 Objet: Re: [ovirt-users] Bad CPU TYPE after Centos 8.3 Hi Lionel, could you please run the following command on the non responsive host? virsh domcapabilities Also, could you please go to the Webadmin UI and check the warning signs on the host list and host detail to see, what flags is the host missing? You can also check the CPU Type on the host's detail page + all available cpus in the info icon next to it. Regards, Lucia

Thank you for the trick put the result as attachment. De: "jb" <jonbae77@gmail.com> À: "users" <users@ovirt.org> Envoyé: Lundi 14 Décembre 2020 15:41:48 Objet: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 Am 14.12.20 um 15:28 schrieb Lionel Caignec: Hi Thank for your answer. the virsh command ask me to authenticate, i tried my ldap account without success. There is an internal account? You can try it with: virsh -c qemu :///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities BQ_BEGIN For the error message no capabibilities is missing, it said bad cpu type but did not change it. "Host CPU type is not compatible with Cluster Properties." Lionel BQ_END _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N7C42VPTYBQ2OX...

Hi Lionel, Thank you for the output from the virsh command. The interesting part is this: <model usable='yes'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model> The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is probably configured with. Could you please run the following query for your cluster to be sure? select name, cpu_name, cpu_flags, cpu_verb from cluster; Could you please also run the virsh domcapabilities command on one of your active and running 8.2 hosts to see if they support Cascadelake-Server-noTSX? 1: https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3 Thank you, Lucia On Mon, Dec 14, 2020 at 7:37 PM <thomas@hoberg.net> wrote:

Hi, that CPU looks rather good to me: KVM/oVirt should love it (as would I)!

I am actually more inclined to believe that it may be a side channel mitigation issue: KVM is distingiuishing between base CPUs and CPUs with enabled mitigations to ensure that you won't accidentally migrate a VM from a 'safe' CPU to one that's vulnerable.

So I suggest you have a look at that, seen what's enabled in BIOS and via microcode updates and check that against what the cluster wants. Perhaps it may involve as little as ensuring all current patches are in (with reboots) after the update to 8.3.

E.g. you might see this when the cluster was previously running on a system with mitigations active, but now mitigations are off (e.g. because the latest microcode updates are not loaded yet).

My personal impression is that it wasn't a terribly good idea to mix mitigations and CPU capabilities, but that they didn't have time/means for a full redesign to what they might have hoped was a singular/one shot issue, before it developed into a whole family of cancers.

Bonne chance! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OHLWLUG23QST3N...

Hi Lionel, in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was dropped in 8.3). The problem you're facing is caused by this transition. In the cluster table you can see that the running hosts still use the old configuration - Cascadelake-Server. That is why the new 8.3 host is non operative. Now the question is why the configuration in the cluster table was not updated. This should be the correct workflow: 1. engine update from 4.4.2 to 4.4.3 2. during engine-setup, the configuration values should be updated in the vdc_table in the database 3. during engine startup, the cluster value should be updated IF all hosts are UP AND all hosts comply to the new configuration Could anything from the below cause the configuration in the cluster table not to be updated? a) engine-setup was not performed b) not all hosts in the cluster were UP c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can see that the 8.2 you've checked does support it, maybe it would be worth checking the others as well). Based on your answer we can try a workaround in the Webadmin before changing the kernel values. Regards, Lucia On Tue, Dec 15, 2020 at 11:01 AM Lionel Caignec <caignec@cines.fr> wrote:

Hi All, and thanks for helping me

So if i does not mistakte, we cannot use anymore cascadelake Server in centos 8.3? Cluster configuration is Secure CascadeLake Server Family

Here is output of the sql query :

name | cpu_name | cpu_flags | cpu_verb

------------+----------------------------------------+----------------------------------------------+----------------------------------------------------------------------------- moria | Secure Intel Cascadelake Server Family | vmx,mds-no,md-clear,model_Cascadelake-Server | Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities

So in your link they said TSX is disabled by default, may i enable it? or the best option is to change cluster cpu type to lower value?

Sorry for all my question i'm quite lost

Regards, Lionel. ------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *thomas@hoberg.net *Cc: *"users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 10:20:08 *Objet: *[ovirt-users] Re: Bad CPU TYPE after Centos 8.3

Hi Lionel,

Thank you for the output from the virsh command. The interesting part is this:

<model usable='yes'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model>

The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is probably configured with.

Could you please run the following query for your cluster to be sure?

select name, cpu_name, cpu_flags, cpu_verb from cluster;

Could you please also run the virsh domcapabilities command on one of your active and running 8.2 hosts to see if they support Cascadelake-Server-noTSX?

1: https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3

Thank you,

Lucia

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PWYZH3SK2WALLM...

Hi Lionel, if you managed to remove the host (or move it to a different cluster) so that all hosts in your cluster are UP - you could try to put one of your running hosts to the maintenance and then back to active - the cluster settings should be updated on the host activation. You can look for the log "Updating cluster CPU flags and verb according to the configuration of the [cluster name]". If that does not help, you can try to restart the engine. Please let me know if that helps. Lucia On Tue, Dec 15, 2020 at 12:53 PM Lionel Caignec <caignec@cines.fr> wrote:

Thank you i understand now.

I double checked all my 8.2 host about cababilities and they seems ok.

I think i foudn the source of this "bug".

For this time i've updated one host in 8.3 (the non operative one), and then i update engine and run engine-setup (with all host up and activated except the 8.3 one). Maybe this is why the cluster table is not updated?

What do you think, if y remove the 8.3 host, rerun "engine-setup" an re-add the 8.3 host?

Lionel.

------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 11:31:47 *Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3

Hi Lionel,

in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was dropped in 8.3).

The problem you're facing is caused by this transition. In the cluster table you can see that the running hosts still use the old configuration - Cascadelake-Server. That is why the new 8.3 host is non operative.

Now the question is why the configuration in the cluster table was not updated. This should be the correct workflow: 1. engine update from 4.4.2 to 4.4.3 2. during engine-setup, the configuration values should be updated in the vdc_table in the database 3. during engine startup, the cluster value should be updated IF all hosts are UP AND all hosts comply to the new configuration

Could anything from the below cause the configuration in the cluster table not to be updated? a) engine-setup was not performed b) not all hosts in the cluster were UP c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can see that the 8.2 you've checked does support it, maybe it would be worth checking the others as well).

Based on your answer we can try a workaround in the Webadmin before changing the kernel values.

Regards,

Lucia

On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec <caignec@cines.fr> wrote:

Hi

thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..."

Good to see that it works for you now (the credit to Lucia). So the cluster configuration didn't change because not all hosts were UP when starting the 4.4.3 engine?

I tried to move guest from 8.2 host to 8.3 one but it failed.

Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).

vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)

The migration probably fails due to incompatible CPU flags - please file a bug and attach VDSM and libvirt logs from both source and destination hosts.

Another think i've noticed on event log of 8.3 host : VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')

But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7

I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.

Lionel.

------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 13:46:11 *Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3

Hi Lionel,

if you managed to remove the host (or move it to a different cluster) so that all hosts in your cluster are UP - you could try to put one of your running hosts to the maintenance and then back to active - the cluster settings should be updated on the host activation. You can look for the log "Updating cluster CPU flags and verb according to the configuration of the [cluster name]".

If that does not help, you can try to restart the engine.

Please let me know if that helps.

Lucia

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YCL3UBDQWSHXKU...

On Tue, Dec 15, 2020 at 4:41 PM Lionel Caignec <caignec@cines.fr> wrote:

Hi Arik,

Yes, my problem about cpu type, seems linked to the fact i've upgraded a host in 8.3 before ugprade ovirt in 4.3...

I've done another test, for my problem of vm migration. I start a guest directly on the 8.3 nodes then it can be migrate from/to 8.2/83. I've also tried to force it to start on a 8.2 nodes then migrate to 8.3 nodes, and it also work.

So it seems a full power cycle on guest resolve issue.

Thanks for checking this - it makes sense that when the VM starts with CPU type and flags that are derived from the new cluster configuration, the issue doesn't happen anymore.

Ok for opening a bug on vdsm, could you please let me know where?

Sure - https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm

Thank you.

------------------------------ *De: *"Arik Hadas" <ahadas@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"Lucia Jelinkova" <ljelinko@redhat.com>, "thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 15:21:22 *Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3

On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec <caignec@cines.fr> wrote:

...

Hi

thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..."

Good to see that it works for you now (the credit to Lucia). So the cluster configuration didn't change because not all hosts were UP when starting the 4.4.3 engine?

...

I tried to move guest from 8.2 host to 8.3 one but it failed.

Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr ' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).

vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)

The migration probably fails due to incompatible CPU flags - please file a bug and attach VDSM and libvirt logs from both source and destination hosts.

...

Another think i've noticed on event log of 8.3 host : VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')

But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7

I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.

Lionel.