2:49 a.m.
Hi,
i've just upgraded one host to latest centos release. And after reboot ovirt said
"Host CPU type is not compatible with Cluster Properties."
Looking on server i can see cpu is detected as skylake (cat
/sys/devices/cpu/caps/pmu_name).
My CPU is Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz so according to ark cascade lake
server.
When i installed my new ovirt environment month agos, i configure cluster to be
"secure Intel Cascade lake server family" and all sound goods.
So anyone can help me?
Environment :
OS : Centos 8.3 (for manager and host in error)
ovirt-engine 4.4.3.12-1.el8
Host with error : vdsm.x86_64 4.40.35.1-1.el8
Working Host : vdsm.x86_64 .40.26.3-1.el8
Maybe someone can help me? I don't know what to do, and does not want to try update
another host if it fail...
Sorry if i post my message at wrong place.
Lionel Caignec.
4:30 a.m.
Hi Lionel,
could you please run the following command on the non responsive host?
virsh domcapabilities
Also, could you please go to the Webadmin UI and check the warning signs on
the host list and host detail to see, what flags is the host missing? You
can also check the CPU Type on the host's detail page + all available cpus
in the info icon next to it.
Regards,
Lucia
On Mon, Dec 14, 2020 at 10:02 AM Lionel Caignec <caignec(a)cines.fr> wrote:
Hi,
i've just upgraded one host to latest centos release. And after reboot
ovirt said "Host CPU type is not compatible with Cluster Properties."
Looking on server i can see cpu is detected as skylake (cat
/sys/devices/cpu/caps/pmu_name).
My CPU is Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz so according to ark
cascade lake server.
When i installed my new ovirt environment month agos, i configure cluster
to be "secure Intel Cascade lake server family" and all sound goods.
So anyone can help me?
Environment :
OS : Centos 8.3 (for manager and host in error)
ovirt-engine 4.4.3.12-1.el8
Host with error : vdsm.x86_64 4.40.35.1-1.el8
Working Host : vdsm.x86_64 .40.26.3-1.el8
Maybe someone can help me? I don't know what to do, and does not want to
try update another host if it fail...
Sorry if i post my message at wrong place.
Lionel Caignec.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N3PPT34GBRL...
8:28 a.m.
Hi
Thank for your answer.
the virsh command ask me to authenticate, i tried my ldap account without success. There
is an internal account?
For the error message no capabibilities is missing, it said bad cpu type but did not
change it.
"Host CPU type is not compatible with Cluster Properties."
Lionel
De: "Lucia Jelinkova" <ljelinko(a)redhat.com>
À: "Lionel Caignec" <caignec(a)cines.fr>
Cc: "users" <users(a)ovirt.org>
Envoyé: Lundi 14 Décembre 2020 11:30:19
Objet: Re: [ovirt-users] Bad CPU TYPE after Centos 8.3
Hi Lionel,
could you please run the following command on the non responsive host?
virsh domcapabilities
Also, could you please go to the Webadmin UI and check the warning signs on the host list
and host detail to see, what flags is the host missing? You can also check the CPU Type on
the host's detail page + all available cpus in the info icon next to it.
Regards,
Lucia
8:41 a.m.
Am 14.12.20 um 15:28 schrieb Lionel Caignec:
Hi
Thank for your answer.
the virsh command ask me to authenticate, i tried my ldap account
without success. There is an internal account?
You can try it with: virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
domcapabilities
For the error message no capabibilities is missing, it said bad cpu
type but did not change it.
"Host CPU type is not compatible with Cluster Properties."
Lionel
------------------------------------------------------------------------
*De: *"Lucia Jelinkova" <ljelinko(a)redhat.com>
*À: *"Lionel Caignec" <caignec(a)cines.fr>
*Cc: *"users" <users(a)ovirt.org>
*Envoyé: *Lundi 14 Décembre 2020 11:30:19
*Objet: *Re: [ovirt-users] Bad CPU TYPE after Centos 8.3
Hi Lionel,
could you please run the following command on the non responsive host?
virsh domcapabilities
Also, could you please go to the Webadmin UI and check the warning
signs on the host list and host detail to see, what flags is the host
missing? You can also check the CPU Type on the host's detail page +
all available cpus in the info icon next to it.
Regards,
Lucia
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PD6FNCIEUIE...
9:05 a.m.
Thank you for the trick put the result as attachment.
De: "jb" <jonbae77(a)gmail.com>
À: "users" <users(a)ovirt.org>
Envoyé: Lundi 14 Décembre 2020 15:41:48
Objet: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Am 14.12.20 um 15:28 schrieb Lionel Caignec:
Hi
Thank for your answer.
the virsh command ask me to authenticate, i tried my ldap account without success. There
is an internal account?
You can try it with: virsh -c qemu
:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf domcapabilities
BQ_BEGIN
For the error message no capabibilities is missing, it said bad cpu type but did not
change it.
"Host CPU type is not compatible with Cluster Properties."
Lionel
BQ_END
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N7C42VPTYBQ...
12:36 p.m.
Hi, that CPU looks rather good to me: KVM/oVirt should love it (as would I)!
I am actually more inclined to believe that it may be a side channel mitigation issue: KVM
is distingiuishing between base CPUs and CPUs with enabled mitigations to ensure that you
won't accidentally migrate a VM from a 'safe' CPU to one that's
vulnerable.
So I suggest you have a look at that, seen what's enabled in BIOS and via microcode
updates and check that against what the cluster wants. Perhaps it may involve as little as
ensuring all current patches are in (with reboots) after the update to 8.3.
E.g. you might see this when the cluster was previously running on a system with
mitigations active, but now mitigations are off (e.g. because the latest microcode updates
are not loaded yet).
My personal impression is that it wasn't a terribly good idea to mix mitigations and
CPU capabilities, but that they didn't have time/means for a full redesign to what
they might have hoped was a singular/one shot issue, before it developed into a whole
family of cancers.
Bonne chance!
3:20 a.m.
Hi Lionel,
Thank you for the output from the virsh command. The interesting part is
this:
<model usable='yes'>Cascadelake-Server-noTSX</model>
<model usable='no'>Cascadelake-Server</model>
The Cascadelake-Server is not usable any more in 8.3 [1] yet that is
what the cluster is probably configured with.
Could you please run the following query for your cluster to be sure?
select name, cpu_name, cpu_flags, cpu_verb from cluster;
Could you please also run the virsh domcapabilities command on one of
your active and running 8.2 hosts to see if they support
Cascadelake-Server-noTSX?
1: https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3
Thank you,
Lucia
On Mon, Dec 14, 2020 at 7:37 PM <thomas(a)hoberg.net> wrote:
Hi, that CPU looks rather good to me: KVM/oVirt should love it (as
would
I)!
I am actually more inclined to believe that it may be a side channel
mitigation issue: KVM is distingiuishing between base CPUs and CPUs with
enabled mitigations to ensure that you won't accidentally migrate a VM from
a 'safe' CPU to one that's vulnerable.
So I suggest you have a look at that, seen what's enabled in BIOS and via
microcode updates and check that against what the cluster wants. Perhaps it
may involve as little as ensuring all current patches are in (with reboots)
after the update to 8.3.
E.g. you might see this when the cluster was previously running on a
system with mitigations active, but now mitigations are off (e.g. because
the latest microcode updates are not loaded yet).
My personal impression is that it wasn't a terribly good idea to mix
mitigations and CPU capabilities, but that they didn't have time/means for
a full redesign to what they might have hoped was a singular/one shot
issue, before it developed into a whole family of cancers.
Bonne chance!
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OHLWLUG23QS...
4:01 a.m.
Hi All,
and thanks for helping me
So if i does not mistakte, we cannot use anymore cascadelake Server in centos 8.3?
Cluster configuration is Secure CascadeLake Server Family
Here is output of the sql query :
name | cpu_name | cpu_flags | cpu_verb
------------+----------------------------------------+----------------------------------------------+-----------------------------------------------------------------------------
moria | Secure Intel Cascadelake Server Family |
vmx,mds-no,md-clear,model_Cascadelake-Server |
Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities
So in your link they said TSX is disabled by default, may i enable it? or the best option
is to change cluster cpu type to lower value?
Sorry for all my question i'm quite lost
Regards,
Lionel.
De: "Lucia Jelinkova" <ljelinko(a)redhat.com>
À: thomas(a)hoberg.net
Cc: "users" <users(a)ovirt.org>
Envoyé: Mardi 15 Décembre 2020 10:20:08
Objet: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
Thank you for the output from the virsh command. The interesting part is this:
<model usable='yes'>Cascadelake-Server-noTSX</model>
<model usable='no'>Cascadelake-Server</model>
The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is
probably configured with.
Could you please run the following query for your cluster to be sure?
select name, cpu_name, cpu_flags, cpu_verb from cluster;
Could you please also run the virsh domcapabilities command on one of your active and
running 8.2 hosts to see if they support Cascadelake-Server-noTSX?
1: [ https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3 |
https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3 ]
Thank you,
Lucia
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PWYZH3SK2WA...
4:31 a.m.
Hi Lionel,
in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to
use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been
changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was
dropped in 8.3).
The problem you're facing is caused by this transition. In the cluster
table you can see that the running hosts still use the old configuration -
Cascadelake-Server. That is why the new 8.3 host is non operative.
Now the question is why the configuration in the cluster table was not
updated. This should be the correct workflow:
1. engine update from 4.4.2 to 4.4.3
2. during engine-setup, the configuration values should be updated in the
vdc_table in the database
3. during engine startup, the cluster value should be updated IF all hosts
are UP AND all hosts comply to the new configuration
Could anything from the below cause the configuration in the cluster table
not to be updated?
a) engine-setup was not performed
b) not all hosts in the cluster were UP
c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can
see that the 8.2 you've checked does support it, maybe it would be worth
checking the others as well).
Based on your answer we can try a workaround in the Webadmin before
changing the kernel values.
Regards,
Lucia
On Tue, Dec 15, 2020 at 11:01 AM Lionel Caignec <caignec(a)cines.fr> wrote:
Hi All,
and thanks for helping me
So if i does not mistakte, we cannot use anymore cascadelake Server in
centos 8.3?
Cluster configuration is Secure CascadeLake Server Family
Here is output of the sql query :
name | cpu_name |
cpu_flags |
cpu_verb
------------+----------------------------------------+----------------------------------------------+-----------------------------------------------------------------------------
moria | Secure Intel Cascadelake Server Family |
vmx,mds-no,md-clear,model_Cascadelake-Server |
Cascadelake-Server,+md-clear,+mds-no,-hle,-rtm,+tsx-ctrl,+arch-capabilities
So in your link they said TSX is disabled by default, may i enable it? or
the best option is to change cluster cpu type to lower value?
Sorry for all my question i'm quite lost
Regards,
Lionel.
------------------------------
*De: *"Lucia Jelinkova" <ljelinko(a)redhat.com>
*À: *thomas(a)hoberg.net
*Cc: *"users" <users(a)ovirt.org>
*Envoyé: *Mardi 15 Décembre 2020 10:20:08
*Objet: *[ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
Thank you for the output from the virsh command. The interesting part is
this:
<model usable='yes'>Cascadelake-Server-noTSX</model>
<model usable='no'>Cascadelake-Server</model>
The Cascadelake-Server is not usable any more in 8.3 [1] yet that is what the cluster is
probably configured with.
Could you please run the following query for your cluster to be sure?
select name, cpu_name, cpu_flags, cpu_verb from cluster;
Could you please also run the virsh domcapabilities command on one of your active and
running 8.2 hosts to see if they support Cascadelake-Server-noTSX?
1: https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-RHEL-8.3
Thank you,
Lucia
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PWYZH3SK2WA...
5:52 a.m.
Thank you i understand now.
I double checked all my 8.2 host about cababilities and they seems ok.
I think i foudn the source of this "bug".
For this time i've updated one host in 8.3 (the non operative one), and then i update
engine and run engine-setup (with all host up and activated except the 8.3 one). Maybe
this is why the cluster table is not updated?
What do you think, if y remove the 8.3 host, rerun "engine-setup" an re-add the
8.3 host?
Lionel.
De: "Lucia Jelinkova" <ljelinko(a)redhat.com>
À: "Lionel Caignec" <caignec(a)cines.fr>
Cc: "thomas" <thomas(a)hoberg.net>, "users"
<users(a)ovirt.org>
Envoyé: Mardi 15 Décembre 2020 11:31:47
Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured to use the
Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has been changed to
Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX was dropped in 8.3).
The problem you're facing is caused by this transition. In the cluster table you can
see that the running hosts still use the old configuration - Cascadelake-Server. That is
why the new 8.3 host is non operative.
Now the question is why the configuration in the cluster table was not updated. This
should be the correct workflow:
1. engine update from 4.4.2 to 4.4.3
2. during engine-setup, the configuration values should be updated in the vdc_table in the
database
3. during engine startup, the cluster value should be updated IF all hosts are UP AND all
hosts comply to the new configuration
Could anything from the below cause the configuration in the cluster table not to be
updated?
a) engine-setup was not performed
b) not all hosts in the cluster were UP
c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I can see that the
8.2 you've checked does support it, maybe it would be worth checking the others as
well).
Based on your answer we can try a workaround in the Webadmin before changing the kernel
values.
Regards,
Lucia
6:46 a.m.
Hi Lionel,
if you managed to remove the host (or move it to a different cluster) so
that all hosts in your cluster are UP - you could try to put one of your
running hosts to the maintenance and then back to active - the cluster
settings should be updated on the host activation. You can look for the log
"Updating cluster CPU flags and verb according to the configuration of the
[cluster name]".
If that does not help, you can try to restart the engine.
Please let me know if that helps.
Lucia
On Tue, Dec 15, 2020 at 12:53 PM Lionel Caignec <caignec(a)cines.fr> wrote:
Thank you i understand now.
I double checked all my 8.2 host about cababilities and they seems ok.
I think i foudn the source of this "bug".
For this time i've updated one host in 8.3 (the non operative one), and
then i update engine and run engine-setup (with all host up and activated
except the 8.3 one). Maybe this is why the cluster table is not updated?
What do you think, if y remove the 8.3 host, rerun "engine-setup" an
re-add the 8.3 host?
Lionel.
------------------------------
*De: *"Lucia Jelinkova" <ljelinko(a)redhat.com>
*À: *"Lionel Caignec" <caignec(a)cines.fr>
*Cc: *"thomas" <thomas(a)hoberg.net>, "users"
<users(a)ovirt.org>
*Envoyé: *Mardi 15 Décembre 2020 11:31:47
*Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
in oVirt 4.4.2 the Secure Intel Cascadelake Server Family was configured
to use the Cascadelake-Server libvirt/qemu CPU. In oVirt 4.4.3, this has
been changed to Cascadelake-Server-noTSX libvirt/qemu CPU (since the TSX
was dropped in 8.3).
The problem you're facing is caused by this transition. In the cluster
table you can see that the running hosts still use the old configuration -
Cascadelake-Server. That is why the new 8.3 host is non operative.
Now the question is why the configuration in the cluster table was not
updated. This should be the correct workflow:
1. engine update from 4.4.2 to 4.4.3
2. during engine-setup, the configuration values should be updated in the
vdc_table in the database
3. during engine startup, the cluster value should be updated IF all hosts
are UP AND all hosts comply to the new configuration
Could anything from the below cause the configuration in the cluster table
not to be updated?
a) engine-setup was not performed
b) not all hosts in the cluster were UP
c) some of the 8.2 hosts do not support the Cascadelake-Server-noTSX (I
can see that the 8.2 you've checked does support it, maybe it would be
worth checking the others as well).
Based on your answer we can try a workaround in the Webadmin before
changing the kernel values.
Regards,
Lucia
7:27 a.m.
Hi
thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation
of 8.2 host i saw the log line "updating cluster CPU..."
I tried to move guest from 8.2 host to 8.3 one but it failed.
Engine.log :
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27)
[] Migration of VM 'pc115dev' to host 'bombur-adm.cines.fr' failed: VM
destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'.
Called from VDS 'dis-adm.cines.fr'
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120),
Migration failed (VM: pc115dev, Source: dis-adm.cines.fr, Destination:
bombur-adm.cines.fr).
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13)
[] Migration of VM 'pc115dev' to host 'bombur-adm.cines.fr' failed: VM
destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM
'4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS 'dis-adm.cines.fr'
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source:
dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
vdsm.log (target host) :
WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184)
WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity
(API:1349)
WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e')
Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à
l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
Another think i've noticed on event log of 8.3 host :
VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM', path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
But file seems to exist on host :
ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
-> ../dm-7
I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.
Lionel.
De: "Lucia Jelinkova" <ljelinko(a)redhat.com>
À: "Lionel Caignec" <caignec(a)cines.fr>
Cc: "thomas" <thomas(a)hoberg.net>, "users"
<users(a)ovirt.org>
Envoyé: Mardi 15 Décembre 2020 13:46:11
Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
if you managed to remove the host (or move it to a different cluster) so that all hosts in
your cluster are UP - you could try to put one of your running hosts to the maintenance
and then back to active - the cluster settings should be updated on the host activation.
You can look for the log "Updating cluster CPU flags and verb according to the
configuration of the [cluster name]".
If that does not help, you can try to restart the engine.
Please let me know if that helps.
Lucia
8:21 a.m.
On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec <caignec(a)cines.fr> wrote:
Hi
thank you it's work now. All my host 8.2/8.3 are up and running. During
reactivation of 8.2 host i saw the log line "updating cluster CPU..."
Good to see that it works for you now (the credit to Lucia).
So the cluster configuration didn't change because not all hosts were UP
when starting the 4.4.3 engine?
I tried to move guest from 8.2 host to 8.3 one but it failed.
Engine.log :
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host '
bombur-adm.cines.fr' failed: VM destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(ForkJoinPool-1-worker-19) [] Rerun VM
'4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS
'dis-adm.cines.fr'
ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev,
Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host '
bombur-adm.cines.fr' failed: VM destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) []
Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS '
dis-adm.cines.fr'
ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev,
Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
vdsm.log (target host) :
WARN (Reactor thread) [vds.dispatcher] unhandled write event
(betterAsyncore:184)
WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and
confirmConnectivity (API:1349)
WARN (vm/4fd61e41) [virt.vm]
(vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming VM:
Domaine non trouvé : aucun domaine ne correspond à l'UUID
'4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
The migration probably fails due to incompatible CPU flags - please file a
bug and attach VDSM and libvirt logs from both source and destination hosts.
Another think i've noticed on event log of 8.3 host :
VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM',
path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
But file seems to exist on host :
ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
lrwxrwxrwx. 1 root root 7 Dec 15 14:14
/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7
I tried to reboot host & restart engine, set my 8.3 host as SPM ... same
error.
Lionel.
------------------------------
*De: *"Lucia Jelinkova" <ljelinko(a)redhat.com>
*À: *"Lionel Caignec" <caignec(a)cines.fr>
*Cc: *"thomas" <thomas(a)hoberg.net>, "users"
<users(a)ovirt.org>
*Envoyé: *Mardi 15 Décembre 2020 13:46:11
*Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
Hi Lionel,
if you managed to remove the host (or move it to a different cluster) so
that all hosts in your cluster are UP - you could try to put one of your
running hosts to the maintenance and then back to active - the cluster
settings should be updated on the host activation. You can look for the log
"Updating cluster CPU flags and verb according to the configuration of the
[cluster name]".
If that does not help, you can try to restart the engine.
Please let me know if that helps.
Lucia
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YCL3UBDQWSH...
8:41 a.m.
Hi Arik,
Yes, my problem about cpu type, seems linked to the fact i've upgraded a host in 8.3
before ugprade ovirt in 4.3...
I've done another test, for my problem of vm migration.
I start a guest directly on the 8.3 nodes then it can be migrate from/to 8.2/83.
I've also tried to force it to start on a 8.2 nodes then migrate to 8.3 nodes, and it
also work.
So it seems a full power cycle on guest resolve issue.
Ok for opening a bug on vdsm, could you please let me know where?
Thank you.
De: "Arik Hadas" <ahadas(a)redhat.com>
À: "Lionel Caignec" <caignec(a)cines.fr>
Cc: "Lucia Jelinkova" <ljelinko(a)redhat.com>, "thomas"
<thomas(a)hoberg.net>, "users" <users(a)ovirt.org>
Envoyé: Mardi 15 Décembre 2020 15:21:22
Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec < [ mailto:caignec@cines.fr |
caignec(a)cines.fr ] > wrote:
Hi
thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation
of 8.2 host i saw the log line "updating cluster CPU..."
Good to see that it works for you now (the credit to Lucia).
So the cluster configuration didn't change because not all hosts were UP when starting
the 4.4.3 engine?
BQ_BEGIN
I tried to move guest from 8.2 host to 8.3 one but it failed.
Engine.log :
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27)
[] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ |
bombur-adm.cines.fr ] ' failed: VM destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'.
Called from VDS ' [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] '
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120),
Migration failed (VM: pc115dev, Source: [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] ,
Destination: [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ).
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13)
[] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ |
bombur-adm.cines.fr ] ' failed: VM destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM
'4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' [
http://dis-adm.cines.fr/ | dis-adm.cines.fr ] '
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: [
http://dis-adm.cines.fr/ | dis-adm.cines.fr ] , Destination: [ http://bombur-adm.cines.fr/
| bombur-adm.cines.fr ] ).
vdsm.log (target host) :
WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184)
WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity
(API:1349)
WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e')
Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à
l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
BQ_END
The migration probably fails due to incompatible CPU flags - please file a bug and attach
VDSM and libvirt logs from both source and destination hosts.
BQ_BEGIN
Another think i've noticed on event log of 8.3 host :
VDSM [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] command SpmStatusVDS failed:
Cannot inquire Lease(name='SDM',
path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2,
'Sanlock get hosts failure', 'No such file or directory')
But file seems to exist on host :
ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
-> ../dm-7
I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.
Lionel.
BQ_END
8:50 a.m.
On Tue, Dec 15, 2020 at 4:41 PM Lionel Caignec <caignec(a)cines.fr> wrote:
Hi Arik,
Yes, my problem about cpu type, seems linked to the fact i've upgraded a
host in 8.3 before ugprade ovirt in 4.3...
I've done another test, for my problem of vm migration.
I start a guest directly on the 8.3 nodes then it can be migrate from/to
8.2/83.
I've also tried to force it to start on a 8.2 nodes then migrate to 8.3
nodes, and it also work.
So it seems a full power cycle on guest resolve issue.
Thanks for checking this - it makes sense that when the VM starts with CPU
type and flags that are derived from the new cluster configuration, the
issue doesn't happen anymore.
Ok for opening a bug on vdsm, could you please let me know where?
Sure - https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
Thank you.
------------------------------
*De: *"Arik Hadas" <ahadas(a)redhat.com>
*À: *"Lionel Caignec" <caignec(a)cines.fr>
*Cc: *"Lucia Jelinkova" <ljelinko(a)redhat.com>, "thomas"
<thomas(a)hoberg.net>,
"users" <users(a)ovirt.org>
*Envoyé: *Mardi 15 Décembre 2020 15:21:22
*Objet: *Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec <caignec(a)cines.fr> wrote:
> Hi
>
> thank you it's work now. All my host 8.2/8.3 are up and running. During
> reactivation of 8.2 host i saw the log line "updating cluster CPU..."
>
Good to see that it works for you now (the credit to Lucia).
So the cluster configuration didn't change because not all hosts were UP
when starting the 4.4.3 engine?
>
> I tried to move guest from 8.2 host to 8.3 one but it failed.
>
>
> Engine.log :
> ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-27) [] Migration of VM 'pc115dev' to host '
> bombur-adm.cines.fr' failed: VM destroyed during the startup.
> ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
> (ForkJoinPool-1-worker-19) [] Rerun VM
> '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS
'dis-adm.cines.fr
> '
> ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID:
> VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev,
> Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
> ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
> (ForkJoinPool-1-worker-13) [] Migration of VM 'pc115dev' to host '
> bombur-adm.cines.fr' failed: VM destroyed during the startup.
> ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) []
> Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS '
> dis-adm.cines.fr'
> ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID:
> VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev,
> Source: dis-adm.cines.fr, Destination: bombur-adm.cines.fr).
>
>
> vdsm.log (target host) :
> WARN (Reactor thread) [vds.dispatcher] unhandled write event
> (betterAsyncore:184)
> WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and
> confirmConnectivity (API:1349)
> WARN (vm/4fd61e41) [virt.vm]
> (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e') Couldn't destroy incoming
VM:
> Domaine non trouvé : aucun domaine ne correspond à l'UUID
> '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
>
The migration probably fails due to incompatible CPU flags - please file a
bug and attach VDSM and libvirt logs from both source and destination hosts.
>
> Another think i've noticed on event log of 8.3 host :
> VDSM bombur-adm.cines.fr command SpmStatusVDS failed: Cannot inquire
> Lease(name='SDM',
path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases',
> offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
> directory')
>
>
> But file seems to exist on host :
> ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
> lrwxrwxrwx. 1 root root 7 Dec 15 14:14
> /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases -> ../dm-7
>
>
> I tried to reboot host & restart engine, set my 8.3 host as SPM ... same
> error.
>
>
> Lionel.
>
9:45 a.m.
Ok it's done bugzilla 1907973, i found this message
"Guest CPU doesn't match specification: missing features: tsx-ctrl
(migration:294)" it's seems you're theory is good.
De: "Arik Hadas" <ahadas(a)redhat.com>
À: "Lionel Caignec" <caignec(a)cines.fr>
Cc: "users" <users(a)ovirt.org>
Envoyé: Mardi 15 Décembre 2020 15:50:26
Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
On Tue, Dec 15, 2020 at 4:41 PM Lionel Caignec < [ mailto:caignec@cines.fr |
caignec(a)cines.fr ] > wrote:
Hi Arik,
Yes, my problem about cpu type, seems linked to the fact i've upgraded a host in 8.3
before ugprade ovirt in 4.3...
I've done another test, for my problem of vm migration.
I start a guest directly on the 8.3 nodes then it can be migrate from/to 8.2/83.
I've also tried to force it to start on a 8.2 nodes then migrate to 8.3 nodes, and it
also work.
BQ_BEGIN
So it seems a full power cycle on guest resolve issue.
BQ_END
Thanks for checking this - it makes sense that when the VM starts with CPU type and flags
that are derived from the new cluster configuration, the issue doesn't happen anymore.
BQ_BEGIN
Ok for opening a bug on vdsm, could you please let me know where?
BQ_END
Sure - [ https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm |
https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm ]
BQ_BEGIN
Thank you.
De: "Arik Hadas" < [ mailto:ahadas@redhat.com | ahadas(a)redhat.com ] >
À: "Lionel Caignec" < [ mailto:caignec@cines.fr | caignec(a)cines.fr ] >
Cc: "Lucia Jelinkova" < [ mailto:ljelinko@redhat.com | ljelinko(a)redhat.com ]
>, "thomas" < [ mailto:thomas@hoberg.net | thomas(a)hoberg.net ] >,
"users" < [ mailto:users@ovirt.org | users(a)ovirt.org ] >
Envoyé: Mardi 15 Décembre 2020 15:21:22
Objet: Re: [ovirt-users] Re: Bad CPU TYPE after Centos 8.3
On Tue, Dec 15, 2020 at 3:29 PM Lionel Caignec < [ mailto:caignec@cines.fr |
caignec(a)cines.fr ] > wrote:
BQ_BEGIN
Hi
thank you it's work now. All my host 8.2/8.3 are up and running. During reactivation
of 8.2 host i saw the log line "updating cluster CPU..."
BQ_END
Good to see that it works for you now (the credit to Lucia).
So the cluster configuration didn't change because not all hosts were UP when starting
the 4.4.3 engine?
BQ_BEGIN
I tried to move guest from 8.2 host to 8.3 one but it failed.
Engine.log :
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27)
[] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ |
bombur-adm.cines.fr ] ' failed: VM destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(ForkJoinPool-1-worker-19) [] Rerun VM '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'.
Called from VDS ' [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] '
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-4) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120),
Migration failed (VM: pc115dev, Source: [ http://dis-adm.cines.fr/ | dis-adm.cines.fr ] ,
Destination: [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] ).
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13)
[] Migration of VM 'pc115dev' to host ' [ http://bombur-adm.cines.fr/ |
bombur-adm.cines.fr ] ' failed: VM destroyed during the startup.
ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-39) [] Rerun VM
'4fd61e41-ea48-4ba8-b78f-dba5acb87c9e'. Called from VDS ' [
http://dis-adm.cines.fr/ | dis-adm.cines.fr ] '
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-38) [] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: pc115dev, Source: [
http://dis-adm.cines.fr/ | dis-adm.cines.fr ] , Destination: [ http://bombur-adm.cines.fr/
| bombur-adm.cines.fr ] ).
vdsm.log (target host) :
WARN (Reactor thread) [vds.dispatcher] unhandled write event (betterAsyncore:184)
WARN (jsonrpc/7) [root] ping was deprecated in favor of ping2 and confirmConnectivity
(API:1349)
WARN (vm/4fd61e41) [virt.vm] (vmId='4fd61e41-ea48-4ba8-b78f-dba5acb87c9e')
Couldn't destroy incoming VM: Domaine non trouvé : aucun domaine ne correspond à
l'UUID '4fd61e41-ea48-4ba8-b78f-dba5acb87c9e' (vm:3738)
BQ_END
The migration probably fails due to incompatible CPU flags - please file a bug and attach
VDSM and libvirt logs from both source and destination hosts.
BQ_BEGIN
Another think i've noticed on event log of 8.3 host :
VDSM [ http://bombur-adm.cines.fr/ | bombur-adm.cines.fr ] command SpmStatusVDS failed:
Cannot inquire Lease(name='SDM',
path='/dev/7586152d-338c-415d-93f4-74efd09c02fa/leases', offset=1048576): (2,
'Sanlock get hosts failure', 'No such file or directory')
But file seems to exist on host :
ls -l /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
lrwxrwxrwx. 1 root root 7 Dec 15 14:14 /dev/7586152d-338c-415d-93f4-74efd09c02fa/leases
-> ../dm-7
I tried to reboot host & restart engine, set my 8.3 host as SPM ... same error.
Lionel.
BQ_END
BQ_END
1234
days inactive
1235
days old
15 comments
5 participants
participants (5)
-
Arik Hadas -
jb -
Lionel Caignec -
Lucia Jelinkova -
thomas＠hoberg.net