Bad CPU TYPE after Centos 8.3
Hi, i've just upgraded one host to latest centos release. And after reboot ovirt said "Host CPU type is not compatible with Cluster Properties." Looking on server i can see cpu is detected as skylake (cat /sys/devices/cpu/caps/pmu_name). My CPU is Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz so according to ark cascade lake server. When i installed my new ovirt environment month agos, i configure cluster to be "secure Intel Cascade lake server family" and all sound goods. So anyone can help me? Environment : OS : Centos 8.3 (for manager and host in error) ovirt-engine 4.4.3.12-1.el8 Host with error : vdsm.x86_64 4.40.35.1-1.el8 Working Host : vdsm.x86_64 .40.26.3-1.el8 Maybe someone can help me? I don't know what to do, and does not want to try update another host if it fail... Sorry if i post my message at wrong place. Lionel Caignec.
Hi Lionel, could you please run the following command on the non responsive host? virsh domcapabilities Also, could you please go to the Webadmin UI and check the warning signs on the host list and host detail to see, what flags is the host missing? You can also check the CPU Type on the host's detail page + all available cpus in the info icon next to it. Regards, Lucia On Mon, Dec 14, 2020 at 10:02 AM Lionel Caignec <caignec@cines.fr> wrote:
Hi,
i've just upgraded one host to latest centos release. And after reboot ovirt said "Host CPU type is not compatible with Cluster Properties." Looking on server i can see cpu is detected as skylake (cat /sys/devices/cpu/caps/pmu_name).
My CPU is Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz so according to ark cascade lake server. When i installed my new ovirt environment month agos, i configure cluster to be "secure Intel Cascade lake server family" and all sound goods.
So anyone can help me?
Environment : OS : Centos 8.3 (for manager and host in error) ovirt-engine 4.4.3.12-1.el8 Host with error : vdsm.x86_64 4.40.35.1-1.el8
Working Host : vdsm.x86_64 .40.26.3-1.el8
Maybe someone can help me? I don't know what to do, and does not want to try update another host if it fail...
Sorry if i post my message at wrong place.
Lionel Caignec. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N3PPT34GBRLPLT...
Hi Thank for your answer. the virsh command ask me to authenticate, i tried my ldap account without success. There is an internal account? For the error message no capabibilities is missing, it said bad cpu type but did not change it. "Host CPU type is not compatible with Cluster Properties." Lionel De: "Lucia Jelinkova" <ljelinko@redhat.com> À: "Lionel Caignec" <caignec@cines.fr> Cc: "users" <users@ovirt.org> Envoyé: Lundi 14 Décembre 2020 11:30:19 Objet: Re: [ovirt-users] Bad CPU TYPE after Centos 8.3 Hi Lionel, could you please run the following command on the non responsive host? virsh domcapabilities Also, could you please go to the Webadmin UI and check the warning signs on the host list and host detail to see, what flags is the host missing? You can also check the CPU Type on the host's detail page + all available cpus in the info icon next to it. Regards, Lucia
Am 14.12.20 um 15:28 schrieb Lionel Caignec:
Hi Thank for your answer.
the virsh command ask me to authenticate, i tried my ldap account without success. There is an internal account?
You can try it with: virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities
For the error message no capabibilities is missing, it said bad cpu type but did not change it.
"Host CPU type is not compatible with Cluster Properties."
Lionel
------------------------------------------------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"users" <users@ovirt.org> *Envoyé: *Lundi 14 Décembre 2020 11:30:19 *Objet: *Re: [ovirt-users] Bad CPU TYPE after Centos 8.3
Hi Lionel,
could you please run the following command on the non responsive host?
virsh domcapabilities
Also, could you please go to the Webadmin UI and check the warning signs on the host list and host detail to see, what flags is the host missing? You can also check the CPU Type on the host's detail page + all available cpus in the info icon next to it.
Regards,
Lucia
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PD6FNCIEUIE54J...
Thank you for the trick put the result as attachment. De: "jb" <jonbae77@gmail.com> À: "users" <users@ovirt.org> Envoyé: Lundi 14 Décembre 2020 15:41:48 Objet: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 Am 14.12.20 um 15:28 schrieb Lionel Caignec: Hi Thank for your answer. the virsh command ask me to authenticate, i tried my ldap account without success. There is an internal account? You can try it with: virsh -c qemu :///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities BQ_BEGIN For the error message no capabibilities is missing, it said bad cpu type but did not change it. "Host CPU type is not compatible with Cluster Properties." Lionel BQ_END _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N7C42VPTYBQ2OX...
Hi, that CPU looks rather good to me: KVM/oVirt should love it (as would I)! I am actually more inclined to believe that it may be a side channel mitigation issue: KVM is distingiuishing between base CPUs and CPUs with enabled mitigations to ensure that you won't accidentally migrate a VM from a 'safe' CPU to one that's vulnerable. So I suggest you have a look at that, seen what's enabled in BIOS and via microcode updates and check that against what the cluster wants. Perhaps it may involve as little as ensuring all current patches are in (with reboots) after the update to 8.3. E.g. you might see this when the cluster was previously running on a system with mitigations active, but now mitigations are off (e.g. because the latest microcode updates are not loaded yet). My personal impression is that it wasn't a terribly good idea to mix mitigations and CPU capabilities, but that they didn't have time/means for a full redesign to what they might have hoped was a singular/one shot issue, before it developed into a whole family of cancers. Bonne chance!
Hi Lionel, Thank you for the output from the virsh command. The interesting part is this: <model usable='yes'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model> The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is probably configured with. Could you please run the following query for your cluster to be sure? select name, cpu_name, cpu_flags, cpu_verb from cluster; Could you please also run the virsh domcapabilities command on one of your active and running 8.2 hosts to see if they support Cascadelake-Server-noTSX? 1: https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3 Thank you, Lucia On Mon, Dec 14, 2020 at 7:37 PM <thomas@hoberg.net> wrote:
Hi, that CPU looks rather good to me: KVM/oVirt should love it (as would I)!
I am actually more inclined to believe that it may be a side channel mitigation issue: KVM is distingiuishing between base CPUs and CPUs with enabled mitigations to ensure that you won't accidentally migrate a VM from a 'safe' CPU to one that's vulnerable.
So I suggest you have a look at that, seen what's enabled in BIOS and via microcode updates and check that against what the cluster wants. Perhaps it may involve as little as ensuring all current patches are in (with reboots) after the update to 8.3.
E.g. you might see this when the cluster was previously running on a system with mitigations active, but now mitigations are off (e.g. because the latest microcode updates are not loaded yet).
My personal impression is that it wasn't a terribly good idea to mix mitigations and CPU capabilities, but that they didn't have time/means for a full redesign to what they might have hoped was a singular/one shot issue, before it developed into a whole family of cancers.
Bonne chance! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OHLWLUG23QST3N...
Hi All, and thanks for helping me So if i does not mistakte, we cannot use anymore cascadelake Server in centos 8.3? Cluster configuration is Secure CascadeLake Server Family Here is output of the sql query : name | cpu_name | cpu_flags | cpu_verb ------------+----------------------------------------+----------------------------------------------+----------------------------------------------------------------------------- moria | Secure Intel Cascadelake Server Family | vmx,mds-no,md-clear,model_Cascadelake-Server | Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities So in your link they said TSX is disabled by default, may i enable it? or the best option is to change cluster cpu type to lower value? Sorry for all my question i'm quite lost Regards, Lionel. De: "Lucia Jelinkova" <ljelinko@redhat.com> À: thomas@hoberg.net Cc: "users" <users@ovirt.org> Envoyé: Mardi 15 Décembre 2020 10:20:08 Objet: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 Hi Lionel, Thank you for the output from the virsh command. The interesting part is this: <model usable='yes'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model> The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is probably configured with. Could you please run the following query for your cluster to be sure? select name, cpu_name, cpu_flags, cpu_verb from cluster; Could you please also run the virsh domcapabilities command on one of your active and running 8.2 hosts to see if they support Cascadelake-Server-noTSX? 1: [ https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3 | https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3 ] Thank you, Lucia _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PWYZH3SK2WALLM...
Hi Lionel, in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was dropped in 8.3). The problem you're facing is caused by this transition. In the cluster table you can see that the running hosts still use the old configuration - Cascadelake-Server. That is why the new 8.3 host is non operative. Now the question is why the configuration in the cluster table was not updated. This should be the correct workflow: 1. engine update from 4.4.2 to 4.4.3 2. during engine-setup, the configuration values should be updated in the vdc_table in the database 3. during engine startup, the cluster value should be updated IF all hosts are UP AND all hosts comply to the new configuration Could anything from the below cause the configuration in the cluster table not to be updated? a) engine-setup was not performed b) not all hosts in the cluster were UP c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can see that the 8.2 you've checked does support it, maybe it would be worth checking the others as well). Based on your answer we can try a workaround in the Webadmin before changing the kernel values. Regards, Lucia On Tue, Dec 15, 2020 at 11:01 AM Lionel Caignec <caignec@cines.fr> wrote:
Hi All, and thanks for helping me
So if i does not mistakte, we cannot use anymore cascadelake Server in centos 8.3? Cluster configuration is Secure CascadeLake Server Family
Here is output of the sql query :
name | cpu_name | cpu_flags | cpu_verb
------------+----------------------------------------+----------------------------------------------+----------------------------------------------------------------------------- moria | Secure Intel Cascadelake Server Family | vmx,mds-no,md-clear,model_Cascadelake-Server | Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities
So in your link they said TSX is disabled by default, may i enable it? or the best option is to change cluster cpu type to lower value?
Sorry for all my question i'm quite lost
Regards, Lionel. ------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *thomas@hoberg.net *Cc: *"users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 10:20:08 *Objet: *[ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
Thank you for the output from the virsh command. The interesting part is this:
<model usable='yes'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model>
The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is probably configured with.
Could you please run the following query for your cluster to be sure?
select name, cpu_name, cpu_flags, cpu_verb from cluster;
Could you please also run the virsh domcapabilities command on one of your active and running 8.2 hosts to see if they support Cascadelake-Server-noTSX?
1: https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3
Thank you,
Lucia
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PWYZH3SK2WALLM...
Thank you i understand now. I double checked all my 8.2 host about cababilities and they seems ok. I think i foudn the source of this "bug". For this time i've updated one host in 8.3 (the non operative one), and then i update engine and run engine-setup (with all host up and activated except the 8.3 one). Maybe this is why the cluster table is not updated? What do you think, if y remove the 8.3 host, rerun "engine-setup" an re-add the 8.3 host? Lionel. De: "Lucia Jelinkova" <ljelinko@redhat.com> À: "Lionel Caignec" <caignec@cines.fr> Cc: "thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> Envoyé: Mardi 15 Décembre 2020 11:31:47 Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 Hi Lionel, in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was dropped in 8.3). The problem you're facing is caused by this transition. In the cluster table you can see that the running hosts still use the old configuration - Cascadelake-Server. That is why the new 8.3 host is non operative. Now the question is why the configuration in the cluster table was not updated. This should be the correct workflow: 1. engine update from 4.4.2 to 4.4.3 2. during engine-setup, the configuration values should be updated in the vdc_table in the database 3. during engine startup, the cluster value should be updated IF all hosts are UP AND all hosts comply to the new configuration Could anything from the below cause the configuration in the cluster table not to be updated? a) engine-setup was not performed b) not all hosts in the cluster were UP c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can see that the 8.2 you've checked does support it, maybe it would be worth checking the others as well). Based on your answer we can try a workaround in the Webadmin before changing the kernel values. Regards, Lucia
Hi Lionel, if you managed to remove the host (or move it to a different cluster) so that all hosts in your cluster are UP - you could try to put one of your running hosts to the maintenance and then back to active - the cluster settings should be updated on the host activation. You can look for the log "Updating cluster CPU flags and verb according to the configuration of the [cluster name]". If that does not help, you can try to restart the engine. Please let me know if that helps. Lucia On Tue, Dec 15, 2020 at 12:53 PM Lionel Caignec <caignec@cines.fr> wrote:
Thank you i understand now.
I double checked all my 8.2 host about cababilities and they seems ok.
I think i foudn the source of this "bug".
For this time i've updated one host in 8.3 (the non operative one), and then i update engine and run engine-setup (with all host up and activated except the 8.3 one). Maybe this is why the cluster table is not updated?
What do you think, if y remove the 8.3 host, rerun "engine-setup" an re-add the 8.3 host?
Lionel.
------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 11:31:47 *Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was dropped in 8.3).
The problem you're facing is caused by this transition. In the cluster table you can see that the running hosts still use the old configuration - Cascadelake-Server. That is why the new 8.3 host is non operative.
Now the question is why the configuration in the cluster table was not updated. This should be the correct workflow: 1. engine update from 4.4.2 to 4.4.3 2. during engine-setup, the configuration values should be updated in the vdc_table in the database 3. during engine startup, the cluster value should be updated IF all hosts are UP AND all hosts comply to the new configuration
Could anything from the below cause the configuration in the cluster table not to be updated? a) engine-setup was not performed b) not all hosts in the cluster were UP c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can see that the 8.2 you've checked does support it, maybe it would be worth checking the others as well).
Based on your answer we can try a workaround in the Webadmin before changing the kernel values.
Regards,
Lucia
Hi thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..." I tried to move guest from 8.2 host to 8.3 one but it failed. Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host 'bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host 'bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr). vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738) Another think i've noticed on event log of 8.3 host : VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7 I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error. Lionel. De: "Lucia Jelinkova" <ljelinko@redhat.com> À: "Lionel Caignec" <caignec@cines.fr> Cc: "thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> Envoyé: Mardi 15 Décembre 2020 13:46:11 Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 Hi Lionel, if you managed to remove the host (or move it to a different cluster) so that all hosts in your cluster are UP - you could try to put one of your running hosts to the maintenance and then back to active - the cluster settings should be updated on the host activation. You can look for the log "Updating cluster CPU flags and verb according to the configuration of the [cluster name]". If that does not help, you can try to restart the engine. Please let me know if that helps. Lucia
On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec <caignec@cines.fr> wrote:
Hi
thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..."
Good to see that it works for you now (the credit to Lucia). So the cluster configuration didn't change because not all hosts were UP when starting the 4.4.3 engine?
I tried to move guest from 8.2 host to 8.3 one but it failed.
Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
The migration probably fails due to incompatible CPU flags - please file a bug and attach VDSM and libvirt logs from both source and destination hosts.
Another think i've noticed on event log of 8.3 host : VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')
But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7
I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.
Lionel.
------------------------------ *De: *"Lucia Jelinkova" <ljelinko@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 13:46:11 *Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
if you managed to remove the host (or move it to a different cluster) so that all hosts in your cluster are UP - you could try to put one of your running hosts to the maintenance and then back to active - the cluster settings should be updated on the host activation. You can look for the log "Updating cluster CPU flags and verb according to the configuration of the [cluster name]".
If that does not help, you can try to restart the engine.
Please let me know if that helps.
Lucia
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YCL3UBDQWSHXKU...
Hi Arik, Yes, my problem about cpu type, seems linked to the fact i've upgraded a host in 8.3 before ugprade ovirt in 4.3... I've done another test, for my problem of vm migration. I start a guest directly on the 8.3 nodes then it can be migrate from/to 8.2/83. I've also tried to force it to start on a 8.2 nodes then migrate to 8.3 nodes, and it also work. So it seems a full power cycle on guest resolve issue. Ok for opening a bug on vdsm, could you please let me know where? Thank you. De: "Arik Hadas" <ahadas@redhat.com> À: "Lionel Caignec" <caignec@cines.fr> Cc: "Lucia Jelinkova" <ljelinko@redhat.com>, "thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> Envoyé: Mardi 15 Décembre 2020 15:21:22 Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec < [ mailto:caignec@cines.fr | caignec@cines.fr ] > wrote: Hi thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..." Good to see that it works for you now (the credit to Lucia). So the cluster configuration didn't change because not all hosts were UP when starting the 4.4.3 engine? BQ_BEGIN I tried to move guest from 8.2 host to 8.3 one but it failed. Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] ' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] , Destination: [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] ' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] , Destination: [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ). vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738) BQ_END The migration probably fails due to incompatible CPU flags - please file a bug and attach VDSM and libvirt logs from both source and destination hosts. BQ_BEGIN Another think i've noticed on event log of 8.3 host : VDSM [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7 I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error. Lionel. BQ_END
On Tue, Dec 15, 2020 at 4:41 PM Lionel Caignec <caignec@cines.fr> wrote:
Hi Arik,
Yes, my problem about cpu type, seems linked to the fact i've upgraded a host in 8.3 before ugprade ovirt in 4.3...
I've done another test, for my problem of vm migration. I start a guest directly on the 8.3 nodes then it can be migrate from/to 8.2/83. I've also tried to force it to start on a 8.2 nodes then migrate to 8.3 nodes, and it also work.
So it seems a full power cycle on guest resolve issue.
Thanks for checking this - it makes sense that when the VM starts with CPU type and flags that are derived from the new cluster configuration, the issue doesn't happen anymore.
Ok for opening a bug on vdsm, could you please let me know where?
Sure - https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
Thank you.
------------------------------ *De: *"Arik Hadas" <ahadas@redhat.com> *À: *"Lionel Caignec" <caignec@cines.fr> *Cc: *"Lucia Jelinkova" <ljelinko@redhat.com>, "thomas" <thomas@hoberg.net>, "users" <users@ovirt.org> *Envoyé: *Mardi 15 Décembre 2020 15:21:22 *Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec <caignec@cines.fr> wrote:
Hi
thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..."
Good to see that it works for you now (the credit to Lucia). So the cluster configuration didn't change because not all hosts were UP when starting the 4.4.3 engine?
I tried to move guest from 8.2 host to 8.3 one but it failed.
Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr ' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host ' bombur-adm.cines.fr' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' dis-adm.cines.fr' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
The migration probably fails due to incompatible CPU flags - please file a bug and attach VDSM and libvirt logs from both source and destination hosts.
Another think i've noticed on event log of 8.3 host : VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory')
But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7
I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.
Lionel.
Ok it's done bugzilla 1907973, i found this message "Guest CPU doesn't match specification: missing features: tsx-ctrl (migration:294)" it's seems you're theory is good. De: "Arik Hadas" <ahadas@redhat.com> À: "Lionel Caignec" <caignec@cines.fr> Cc: "users" <users@ovirt.org> Envoyé: Mardi 15 Décembre 2020 15:50:26 Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 On Tue, Dec 15, 2020 at 4:41 PM Lionel Caignec < [ mailto:caignec@cines.fr | caignec@cines.fr ] > wrote: Hi Arik, Yes, my problem about cpu type, seems linked to the fact i've upgraded a host in 8.3 before ugprade ovirt in 4.3... I've done another test, for my problem of vm migration. I start a guest directly on the 8.3 nodes then it can be migrate from/to 8.2/83. I've also tried to force it to start on a 8.2 nodes then migrate to 8.3 nodes, and it also work. BQ_BEGIN So it seems a full power cycle on guest resolve issue. BQ_END Thanks for checking this - it makes sense that when the VM starts with CPU type and flags that are derived from the new cluster configuration, the issue doesn't happen anymore. BQ_BEGIN Ok for opening a bug on vdsm, could you please let me know where? BQ_END Sure - [ https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm | https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm ] BQ_BEGIN Thank you. De: "Arik Hadas" < [ mailto:ahadas@redhat.com | ahadas@redhat.com ] > À: "Lionel Caignec" < [ mailto:caignec@cines.fr | caignec@cines.fr ] > Cc: "Lucia Jelinkova" < [ mailto:ljelinko@redhat.com | ljelinko@redhat.com ] >, "thomas" < [ mailto:thomas@hoberg.net | thomas@hoberg.net ] >, "users" < [ mailto:users@ovirt.org | users@ovirt.org ] > Envoyé: Mardi 15 Décembre 2020 15:21:22 Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3 On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec < [ mailto:caignec@cines.fr | caignec@cines.fr ] > wrote: BQ_BEGIN Hi thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation of 8.2 host i saw the log line "updating cluster CPU..." BQ_END Good to see that it works for you now (the credit to Lucia). So the cluster configuration didn't change because not all hosts were UP when starting the 4.4.3 engine? BQ_BEGIN I tried to move guest from 8.2 host to 8.3 one but it failed. Engine.log : ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] ' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] , Destination: [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ). ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ' failed: VM destroyed during the startup. ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] ' ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] , Destination: [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ). vdsm.log (target host) : WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184) WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity (API:1349) WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738) BQ_END The migration probably fails due to incompatible CPU flags - please file a bug and attach VDSM and libvirt logs from both source and destination hosts. BQ_BEGIN Another think i've noticed on event log of 8.3 host : VDSM [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] command SpmStatusVDS failed: Cannot inquire Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or directory') But file seems to exist on host : ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7 I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error. Lionel. BQ_END BQ_END
participants (5)
-
Arik Hadas -
jb -
Lionel Caignec -
Lucia Jelinkova -
thomas＠hoberg.net