OST: vm_run fails for me (basic-suite-master)
Hi everyone, Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm). Can you point me to what I might be missing? I believe I get the same failure even on Fedora. This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: journalctl -b on host0 shows that libvirt complains about NUMA configuration: Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: libvirt version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem < http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated. Thanks, Ondra
Did you follow the instructions on [1] ? Specifically, verifying ' cat /sys/module/kvm_intel/parameters/nested ' gives you 'Y'. [1] http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installation... On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the same failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: libvirt version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem < http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Yes, I stated that in my message. root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] # cat /sys/module/kvm_intel/parameters/nested :( Y On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/parameters/nested ' gives you 'Y'.
[1] http://ovirt-system-tests.readthedocs.io/en/latest/docs/ general/installation.html
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the same failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: libvirt version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem < http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
What is the host CPU you are using? I came across the same error few days ago, but without running OST, I tried running with Lago: fc24 host -> el7 vm -> el7 vm. I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but it always failed on the same error: kvm_put_msrs: Assertion `ret == n' failed. My CPU is Broadwell btw. Milan, any ideas? you think it might be related? Nadav. On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/parameters/nested ' gives you 'Y'.
[1] http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installation...
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the same failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: libvirt version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem <http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
It is a Skylake-H, and I can see it is not mentioned in lago/virt.py. I guess I'll step through the code (as well as other places discovered by 'git grep cpu') and see if I could solve this by adding the Skylake family to _CPU_FAMILIES. Do you have other pointers? Thanks, Ondra On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running OST, I tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but it always failed on the same error: kvm_put_msrs: Assertion `ret == n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/parameters/nested
'
gives you 'Y'.
[1] http://ovirt-system-tests.readthedocs.io/en/latest/docs/ general/installation.html
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux
system
which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the same failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote: popcnt aes
hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: libvirt version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem <http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
qemu-kvm:
/builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
I would first try testing it without OST, because in OST it will pick the CPU via the cluster family(which is controlled in virt.py). You can try specifying the 'cpu_model' in the init file, skipping the 'cpu family' logic, something like:
cat LagoInitFile domains: vm-el73: memory: 2048 service_provider: systemd cpu_model: Broadwell nics: - net: lago disks: - template_name: el7.3-base type: template name: root dev: vda format: qcow2 nets: lago: type: nat dhcp: start: 100 end: 254 management: true dns_domain_name: lago.local
lago init && lago start
Then install lago again in the VM, copy the same init file, and check if for different combinations of cpu_model it works for you - would give us a hint how to solve this. The 'cpu_model' basically translates to this xml definition in libvirt: <cpu mode='custom' match='exact'> <model fallback='allow'>Broadwell</model> <topology sockets='2' cores='1' threads='1'/> <feature policy='optional' name='vmx'/> <feature policy='optional' name='svm'/> </cpu> I tried manually editing it also to host-passthrough, but still failed on the same error. The thing is that the 'kvm_put_msrs: Assertion `ret == n' failed.' error doesn't give any indication where it failed(or if the cpu is missing a flag), maybe there is a way to debug this at qemu/kvm level? I'm not sure. On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
It is a Skylake-H, and I can see it is not mentioned in lago/virt.py.
I guess I'll step through the code (as well as other places discovered by 'git grep cpu') and see if I could solve this by adding the Skylake family to _CPU_FAMILIES.
Do you have other pointers?
Thanks, Ondra
On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running OST, I tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but it always failed on the same error: kvm_put_msrs: Assertion `ret == n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/parameters/nested ' gives you 'Y'.
[1]
http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installation...
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the same failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: libvirt version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem <http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
In my case, simply adding Skylake-Client a supported CPU family did the trick: https://github.com/lago-project/lago/pull/448 i wonder if Westmere is a good fallback -- it works for you on Broadwell, right? On Wed, Feb 8, 2017 at 1:58 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
I would first try testing it without OST, because in OST it will pick the CPU via the cluster family(which is controlled in virt.py). You can try specifying the 'cpu_model' in the init file, skipping the 'cpu family' logic, something like:
cat LagoInitFile domains: vm-el73: memory: 2048 service_provider: systemd cpu_model: Broadwell nics: - net: lago disks: - template_name: el7.3-base type: template name: root dev: vda format: qcow2 nets: lago: type: nat dhcp: start: 100 end: 254 management: true dns_domain_name: lago.local
lago init && lago start
Then install lago again in the VM, copy the same init file, and check if for different combinations of cpu_model it works for you - would give us a hint how to solve this. The 'cpu_model' basically translates to this xml definition in libvirt: <cpu mode='custom' match='exact'> <model fallback='allow'>Broadwell</model> <topology sockets='2' cores='1' threads='1'/> <feature policy='optional' name='vmx'/> <feature policy='optional' name='svm'/> </cpu>
I tried manually editing it also to host-passthrough, but still failed on the same error. The thing is that the 'kvm_put_msrs: Assertion `ret == n' failed.' error doesn't give any indication where it failed(or if the cpu is missing a flag), maybe there is a way to debug this at qemu/kvm level? I'm not sure.
It is a Skylake-H, and I can see it is not mentioned in lago/virt.py.
I guess I'll step through the code (as well as other places discovered by 'git grep cpu') and see if I could solve this by adding the Skylake family to _CPU_FAMILIES.
Do you have other pointers?
Thanks, Ondra
On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running OST, I tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but it always failed on the same error: kvm_put_msrs: Assertion `ret == n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/
' gives you 'Y'.
[1]
http://ovirt-system-tests.readthedocs.io/en/latest/docs/ general/installation.html
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the
same
failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]:
version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem <http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org ) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt:
2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt:
On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote: parameters/nested libvirt port port
2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
What happened to Milan' PR from a while ago addressing this exact situation? On 08 Feb 2017, at 16:04, Ondrej Svoboda <osvoboda@redhat.com> wrote: In my case, simply adding Skylake-Client a supported CPU family did the trick: https://github.com/lago-project/lago/pull/448 i wonder if Westmere is a good fallback -- it works for you on Broadwell, right? On Wed, Feb 8, 2017 at 1:58 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
I would first try testing it without OST, because in OST it will pick the CPU via the cluster family(which is controlled in virt.py). You can try specifying the 'cpu_model' in the init file, skipping the 'cpu family' logic, something like:
cat LagoInitFile domains: vm-el73: memory: 2048 service_provider: systemd cpu_model: Broadwell nics: - net: lago disks: - template_name: el7.3-base type: template name: root dev: vda format: qcow2 nets: lago: type: nat dhcp: start: 100 end: 254 management: true dns_domain_name: lago.local
lago init && lago start
Then install lago again in the VM, copy the same init file, and check if for different combinations of cpu_model it works for you - would give us a hint how to solve this. The 'cpu_model' basically translates to this xml definition in libvirt: <cpu mode='custom' match='exact'> <model fallback='allow'>Broadwell</model> <topology sockets='2' cores='1' threads='1'/> <feature policy='optional' name='vmx'/> <feature policy='optional' name='svm'/> </cpu>
I tried manually editing it also to host-passthrough, but still failed on the same error. The thing is that the 'kvm_put_msrs: Assertion `ret == n' failed.' error doesn't give any indication where it failed(or if the cpu is missing a flag), maybe there is a way to debug this at qemu/kvm level? I'm not sure.
It is a Skylake-H, and I can see it is not mentioned in lago/virt.py.
I guess I'll step through the code (as well as other places discovered by 'git grep cpu') and see if I could solve this by adding the Skylake family to _CPU_FAMILIES.
Do you have other pointers?
Thanks, Ondra
On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running OST, I tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but it always failed on the same error: kvm_put_msrs: Assertion `ret == n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/
' gives you 'Y'.
[1]
http://ovirt-system-tests.readthedocs.io/en/latest/docs/ general/installation.html
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Hi everyone,
Even though I have nested virtualization enabled in my Arch Linux system which I use to run OST, vm_run is the first test to fail in 004_basic_sanity (followed by snapshots_merge and suspend_resume_vm).
Can you point me to what I might be missing? I believe I get the
same
failure even on Fedora.
This is what host0's CPU capabilities look like (vmx is there): [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) stepping : 1 microcode : 0x1 cpu MHz : 2711.988 cache size : 16384 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid bogomips : 5423.97 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
journalctl -b on host0 shows that libvirt complains about NUMA configuration:
Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]:
version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem <http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org ) Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt:
2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 left promiscuous mode Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt:
On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote: parameters/nested libvirt port port
2(vnet0) entered disabled state Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: hostname: lago-basic-suite-master-host0.lago.local Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable to read from monitor: Connection reset by peer Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: internal error: qemu unexpectedly closed the monitor: 2017-02-07T11:33:23.058571Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config
qemu-kvm: /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: Assertion `ret == n' failed. Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: <info> [1486467203.1025] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged') [30 10 3] Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now active Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: Machine qemu-1-vm0 terminated.
Thanks, Ondra
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Do you mean https://github.com/lago-project/lago/pull/398 which has been merged for over a month? The second sentence in the PR (below) is contradicted by newer, non-recognized CPUs, such as Skylake. "This patch fixes the problems by selecting a minimum reasonable CPU model for the given hardware platform. Westmere is selected unless older or non-Intel hardware is used." On Thu, Feb 9, 2017 at 4:07 PM, Michal Skrivanek <mskrivan@redhat.com> wrote:
What happened to Milan' PR from a while ago addressing this exact situation?
On 08 Feb 2017, at 16:04, Ondrej Svoboda <osvoboda@redhat.com> wrote:
In my case, simply adding Skylake-Client a supported CPU family did the trick: https://github.com/lago-project/lago/pull/448
i wonder if Westmere is a good fallback -- it works for you on Broadwell, right?
On Wed, Feb 8, 2017 at 1:58 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
I would first try testing it without OST, because in OST it will pick the CPU via the cluster family(which is controlled in virt.py). You can try specifying the 'cpu_model' in the init file, skipping the 'cpu family' logic, something like:
cat LagoInitFile domains: vm-el73: memory: 2048 service_provider: systemd cpu_model: Broadwell nics: - net: lago disks: - template_name: el7.3-base type: template name: root dev: vda format: qcow2 nets: lago: type: nat dhcp: start: 100 end: 254 management: true dns_domain_name: lago.local
lago init && lago start
Then install lago again in the VM, copy the same init file, and check if for different combinations of cpu_model it works for you - would give us a hint how to solve this. The 'cpu_model' basically translates to this xml definition in libvirt: <cpu mode='custom' match='exact'> <model fallback='allow'>Broadwell</model> <topology sockets='2' cores='1' threads='1'/> <feature policy='optional' name='vmx'/> <feature policy='optional' name='svm'/> </cpu>
I tried manually editing it also to host-passthrough, but still failed on the same error. The thing is that the 'kvm_put_msrs: Assertion `ret == n' failed.' error doesn't give any indication where it failed(or if the cpu is missing a flag), maybe there is a way to debug this at qemu/kvm level? I'm not sure.
It is a Skylake-H, and I can see it is not mentioned in lago/virt.py.
I guess I'll step through the code (as well as other places discovered by 'git grep cpu') and see if I could solve this by adding the Skylake family to _CPU_FAMILIES.
Do you have other pointers?
Thanks, Ondra
On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running OST, I tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but it always failed on the same error: kvm_put_msrs: Assertion `ret == n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] #
cat
/sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat /sys/module/kvm_intel/paramete
rs/nested
' gives you 'Y'.
[1]
http://ovirt-system-tests.readthedocs.io/en/latest/docs/gene ral/installation.html
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <osvoboda@redhat.com
wrote: > > Hi everyone, > > Even though I have nested virtualization enabled in my Arch Linux > system > which I use to run OST, vm_run is the first test to fail in > 004_basic_sanity > (followed by snapshots_merge and suspend_resume_vm). > > Can you point me to what I might be missing? I believe I get the same > failure even on Fedora. > > This is what host0's CPU capabilities look like (vmx is there): > [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 44 > model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) > stepping : 1 > microcode : 0x1 > cpu MHz : 2711.988 > cache size : 16384 KB > physical id : 0 > siblings : 1 > core id : 0 > cpu cores : 1 > apicid : 0 > initial apicid : 0 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge > mca > cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc > rep_good > nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic > popcnt aes > hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid > bogomips : 5423.97 > clflush size : 64 > cache_alignment : 64 > address sizes : 40 bits physical, 48 bits virtual > power management: > > journalctl -b on host0 shows that libvirt complains about NUMA > configuration: > > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]:
> version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem > <http://bugs.centos.org>, 2017-01-17-23:37:48, c1bm.rdu2.centos.org) > Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt:
> 2(vnet0) entered disabled state > Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device vnet0 > left > promiscuous mode > Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: ovirtmgmt:
On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <osvoboda@redhat.com> wrote: libvirt port port
> 2(vnet0) entered disabled state > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: > hostname: > lago-basic-suite-master-host0.lago.local > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: Unable > to > read from monitor: Connection reset by peer > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: > internal > error: qemu unexpectedly closed the monitor: > 2017-02-07T11:33:23.058571Z > qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 4 5 6 7 > 8 9 > 10 11 12 13 14 15 > > 2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up to > maxcpus > should be described in NUMA config > > qemu-kvm: > /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: kvm_put_msrs: > Assertion `ret == n' failed. > Feb 07 06:33:23 lago-basic-suite-master-host0 NetworkManager[657]: > <info> > [1486467203.1025] device (vnet0): state change: disconnected -> > unmanaged > (reason 'unmanaged') [30 10 3] > Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 guests now > active > Feb 07 06:33:23 lago-basic-suite-master-host0 systemd-machined[22044]: > Machine qemu-1-vm0 terminated. > > Thanks, > Ondra > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_06533245-DCE0-42FC-8274-6308A4089EF3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii
On 9 Feb 2017, at 16:16, Ondrej Svoboda <osvoboda@redhat.com> wrote: =20 Do you mean https://github.com/lago-project/lago/pull/398 = <https://github.com/lago-project/lago/pull/398> which has been merged = for over a month? =20 The second sentence in the PR (below) is contradicted by newer, = non-recognized CPUs, such as Skylake.
=20 "This patch fixes the problems by selecting a minimum reasonable CPU = model for the given hardware platform. Westmere is selected unless older = or non-Intel hardware is used." =20 On Thu, Feb 9, 2017 at 4:07 PM, Michal Skrivanek <mskrivan@redhat.com = <mailto:mskrivan@redhat.com>> wrote: What happened to Milan' PR from a while ago addressing this exact = situation? =20 On 08 Feb 2017, at 16:04, Ondrej Svoboda <osvoboda@redhat.com = <mailto:osvoboda@redhat.com>> wrote: =20
In my case, simply adding Skylake-Client a supported CPU family did =
=20 i wonder if Westmere is a good fallback -- it works for you on = Broadwell, right? =20 On Wed, Feb 8, 2017 at 1:58 PM, Nadav Goldin <ngoldin@redhat.com = <mailto:ngoldin@redhat.com>> wrote: I would first try testing it without OST, because in OST it will pick the CPU via the cluster family(which is controlled in virt.py). You can try specifying the 'cpu_model' in the init file, skipping the = 'cpu family' logic, something like: =20
cat LagoInitFile domains: vm-el73: memory: 2048 service_provider: systemd cpu_model: Broadwell nics: - net: lago disks: - template_name: el7.3-base type: template name: root dev: vda format: qcow2 nets: lago: type: nat dhcp: start: 100 end: 254 management: true dns_domain_name: lago.local =20 lago init && lago start =20 Then install lago again in the VM, copy the same init file, and check if for different combinations of cpu_model it works for you - would give us a hint how to solve this. The 'cpu_model' basically =
to this xml definition in libvirt: <cpu mode=3D'custom' match=3D'exact'> <model fallback=3D'allow'>Broadwell</model> <topology sockets=3D'2' cores=3D'1' threads=3D'1'/> <feature policy=3D'optional' name=3D'vmx'/> <feature policy=3D'optional' name=3D'svm'/> </cpu> =20 I tried manually editing it also to host-passthrough, but still = failed on the same error. The thing is that the 'kvm_put_msrs: Assertion = `ret =3D=3D n' failed.' error doesn't give any indication where it = failed(or if the cpu is missing a flag), maybe there is a way to debug this at qemu/kvm level? I'm not sure. =20 =20 =20 =20 =20 =20 On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <osvoboda@redhat.com = <mailto:osvoboda@redhat.com>> wrote:
It is a Skylake-H, and I can see it is not mentioned in = lago/virt.py.
I guess I'll step through the code (as well as other places = discovered by 'git grep cpu') and see if I could solve this by adding the Skylake = family to _CPU_FAMILIES.
Do you have other pointers?
Thanks, Ondra
On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com = <mailto:ngoldin@redhat.com>> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running =
OST, I
tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but = it always failed on the same error: kvm_put_msrs: Assertion `ret =3D=3D n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda = <osvoboda@redhat.com <mailto:osvoboda@redhat.com>> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] = # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com = <mailto:eedri@redhat.com>> wrote:
Did you follow the instructions on [1] ?
Specifically, verifying ' cat =
/sys/module/kvm_intel/parameters/nested
' gives you 'Y'.
[1]
= http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installati= on.html = <http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installat= ion.html>
On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda = <osvoboda@redhat.com <mailto:osvoboda@redhat.com>> wrote: > > Hi everyone, > > Even though I have nested virtualization enabled in my Arch = Linux > system > which I use to run OST, vm_run is the first test to fail in > 004_basic_sanity > (followed by snapshots_merge and suspend_resume_vm). > > Can you point me to what I might be missing? I believe I get =
> failure even on Fedora. > > This is what host0's CPU capabilities look like (vmx is =
> [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo > processor : 0 > vendor_id : GenuineIntel > cpu family : 6 > model : 44 > model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) > stepping : 1 > microcode : 0x1 > cpu MHz : 2711.988 > cache size : 16384 KB > physical id : 0 > siblings : 1 > core id : 0 > cpu cores : 1 > apicid : 0 > initial apicid : 0 > fpu : yes > fpu_exception : yes > cpuid level : 11 > wp : yes > flags : fpu vme de pse tsc msr pae mce cx8 apic sep = mtrr pge > mca > cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm = constant_tsc > rep_good > nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 = x2apic > popcnt aes > hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid > bogomips : 5423.97 > clflush size : 64 > cache_alignment : 64 > address sizes : 40 bits physical, 48 bits virtual > power management: > > journalctl -b on host0 shows that libvirt complains about NUMA > configuration: > > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: =
How/why? Westmere should have been selected in that case the trick: https://github.com/lago-project/lago/pull/448 = <https://github.com/lago-project/lago/pull/448> translates the same there): libvirt
> version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem > <http://bugs.centos.org <http://bugs.centos.org/>>, = 2017-01-17-23 <tel:2017-01-17-23>:37:48, c1bm.rdu2.centos.org = <http://c1bm.rdu2.centos.org/>) > Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: = ovirtmgmt: port > 2(vnet0) entered disabled state > Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device = vnet0 > left > promiscuous mode > Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: = ovirtmgmt: port > 2(vnet0) entered disabled state > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: > hostname: > lago-basic-suite-master-host0.lago.local > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: = Unable > to > read from monitor: Connection reset by peer > Feb 07 06:33:23 lago-basic-suite-master-host0 libvirtd[12888]: > internal > error: qemu unexpectedly closed the monitor: > 2017-02-07T11:33:23.058571Z > qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 3 = 4 5 6 7 > 8 9 > 10 11 12 13 14 15 > > 2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up = to > maxcpus > should be described in NUMA config > > qemu-kvm: > /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: = kvm_put_msrs: > Assertion `ret =3D=3D n' failed. > Feb 07 06:33:23 lago-basic-suite-master-host0 = NetworkManager[657]: > <info> > [1486467203.1025] device (vnet0): state change: disconnected = -> > unmanaged > (reason 'unmanaged') [30 10 3] > Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 = guests now > active > Feb 07 06:33:23 lago-basic-suite-master-host0 = systemd-machined[22044]: > Machine qemu-1-vm0 terminated. > > Thanks, > Ondra > > _______________________________________________ > Devel mailing list > Devel@ovirt.org <mailto:Devel@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
-- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel
phone: +972-9-7692018 <tel:%2B972-9-7692018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
=20 _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20
--Apple-Mail=_06533245-DCE0-42FC-8274-6308A4089EF3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 9 Feb 2017, at 16:16, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" class=3D"">osvoboda@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">Do you mean <a = href=3D"https://github.com/lago-project/lago/pull/398" = class=3D"">https://github.com/lago-project/lago/pull/398</a> which has = been merged for over a month?<br class=3D""><br class=3D""></div>The = second sentence in the PR (below) is contradicted by newer, = non-recognized CPUs, such as Skylake.<br = class=3D""></div></div></blockquote><div><br class=3D""></div>How/why? = Westmere should have been selected in that case</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D""><div class=3D""><div class=3D""><br= class=3D"">"This patch fixes the problems by selecting a minimum = reasonable CPU model for the given hardware platform. Westmere is = selected unless older or non-Intel hardware is = used."</div></div></div></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Thu, Feb 9, 2017 at 4:07 PM, = Michal Skrivanek <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:mskrivan@redhat.com" target=3D"_blank" = class=3D"">mskrivan@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto" = class=3D""><div class=3D""></div><div class=3D"">What happened to Milan' = PR from a while ago addressing this exact situation?</div><div = class=3D""><div class=3D"h5"><div class=3D""><br class=3D"">On 08 Feb = 2017, at 16:04, Ondrej Svoboda <<a href=3D"mailto:osvoboda@redhat.com" = target=3D"_blank" class=3D"">osvoboda@redhat.com</a>> wrote:<br = class=3D""><br class=3D""></div><blockquote type=3D"cite" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div class=3D""><div class=3D"">In = my case, simply adding Skylake-Client a supported CPU family did the = trick: <a href=3D"https://github.com/lago-project/lago/pull/448" = target=3D"_blank" class=3D"">https://github.com/lago-<wbr = class=3D"">project/lago/pull/448</a><br class=3D""><br class=3D""></div>i = wonder if Westmere is a good fallback -- it works for you on Broadwell, = right?<br class=3D""></div></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Wed, Feb 8, 2017 at 1:58 PM, = Nadav Goldin <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:ngoldin@redhat.com" target=3D"_blank" = class=3D"">ngoldin@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex">I would first try = testing it without OST, because in OST it will pick<br class=3D""> the CPU via the cluster family(which is controlled in virt.py). You<br = class=3D""> can try specifying the 'cpu_model' in the init file, skipping the = 'cpu<br class=3D""> family' logic, something like:<br class=3D""> <br class=3D""> > cat LagoInitFile<br class=3D""> domains:<br class=3D""> vm-el73:<br class=3D""> memory: 2048<br class=3D""> service_provider: systemd<br class=3D""> cpu_model: Broadwell<br class=3D""> nics:<br class=3D""> - net: lago<br class=3D""> disks:<br class=3D""> - template_name: el7.3-base<br class=3D""> type: template<br class=3D""> name: root<br class=3D""> dev: vda<br class=3D""> format: qcow2<br class=3D""> nets:<br class=3D""> lago:<br class=3D""> type: nat<br class=3D""> dhcp:<br class=3D""> start: 100<br class=3D""> end: 254<br class=3D""> management: true<br class=3D""> dns_domain_name: lago.local<br class=3D""> <br class=3D""> > lago init && lago start<br class=3D""> <br class=3D""> Then install lago again in the VM, copy the same init file, and check<br = class=3D""> if for different combinations of cpu_model it works for you - would<br = class=3D""> give us a hint how to solve this. The 'cpu_model' basically = translates<br class=3D""> to this xml definition in libvirt:<br class=3D""> <cpu mode=3D'custom' match=3D'exact'><br class=3D""> <model fallback=3D'allow'>Broadwell</mo<wbr = class=3D"">del><br class=3D""> <topology sockets=3D'2' cores=3D'1' threads=3D'1'/><br= class=3D""> <feature policy=3D'optional' name=3D'vmx'/><br = class=3D""> <feature policy=3D'optional' name=3D'svm'/><br = class=3D""> </cpu><br class=3D""> <br class=3D""> I tried manually editing it also to host-passthrough, but still = failed<br class=3D""> on the same error. The thing is that the 'kvm_put_msrs: Assertion = `ret<br class=3D""> =3D=3D n' failed.' error doesn't give any indication where it failed(or = if<br class=3D""> the cpu is missing a flag), maybe there is a way to debug this at<br = class=3D""> qemu/kvm level? I'm not sure.<br class=3D""> <div class=3D"m_-909161191278510801HOEnZb"><div = class=3D"m_-909161191278510801h5"><br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" target=3D"_blank" = class=3D"">osvoboda@redhat.com</a>> wrote:<br class=3D""> > It is a Skylake-H, and I can see it is not mentioned in = lago/virt.py.<br class=3D""> ><br class=3D""> > I guess I'll step through the code (as well as other places = discovered by<br class=3D""> > 'git grep cpu') and see if I could solve this by adding the Skylake = family<br class=3D""> > to _CPU_FAMILIES.<br class=3D""> ><br class=3D""> > Do you have other pointers?<br class=3D""> ><br class=3D""> > Thanks,<br class=3D""> > Ondra<br class=3D""> ><br class=3D""> > On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <<a = href=3D"mailto:ngoldin@redhat.com" target=3D"_blank" = class=3D"">ngoldin@redhat.com</a>> wrote:<br class=3D""> >><br class=3D""> >> What is the host CPU you are using?<br class=3D""> >> I came across the same error few days ago, but without running = OST, I<br class=3D""> >> tried running with Lago:<br class=3D""> >> fc24 host -> el7 vm -> el7 vm.<br class=3D""> >><br class=3D""> >> I have a slight suspect that it is related to the CPU model = we<br class=3D""> >> configure in libvirt, I tried a mixture of few<br class=3D""> >> combinations(host-pass-through<wbr class=3D"">, pinning down = the CPU model), but it<br class=3D""> >> always failed on the same error:<br class=3D""> >> kvm_put_msrs: Assertion `ret =3D=3D n' failed.<br class=3D""> >><br class=3D""> >> My CPU is Broadwell btw.<br class=3D""> >><br class=3D""> >><br class=3D""> >> Milan, any ideas? you think it might be related?<br class=3D""> >><br class=3D""> >> Nadav.<br class=3D""> >><br class=3D""> >><br class=3D""> >><br class=3D""> >> On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" target=3D"_blank" = class=3D"">osvoboda@redhat.com</a>><br class=3D""> >> wrote:<br class=3D""> >> > Yes, I stated that in my message.<br class=3D""> >> ><br class=3D""> >> > root@osvoboda-t460p /home/src/ovirt-system-tests = (git)-[master] # cat<br class=3D""> >> > /sys/module/kvm_intel/paramete<wbr class=3D"">rs/nested<br = class=3D""> >> > :(<br class=3D""> >> > Y<br class=3D""> >> ><br class=3D""> >> > On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D"">eedri@redhat.com</a>> wrote:<br class=3D""> >> >><br class=3D""> >> >> Did you follow the instructions on [1] ?<br class=3D""> >> >><br class=3D""> >> >> Specifically, verifying ' cat = /sys/module/kvm_intel/paramete<wbr class=3D"">rs/nested<br class=3D""> >> >> '<br class=3D""> >> >> gives you 'Y'.<br class=3D""> >> >><br class=3D""> >> >> [1]<br class=3D""> >> >><br class=3D""> >> >> <a = href=3D"http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/in= stallation.html" rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://ovirt-system-tests.read<wbr = class=3D"">thedocs.io/en/latest/docs/gene<wbr = class=3D"">ral/installation.html</a><br class=3D""> >> >><br class=3D""> >> >> On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" target=3D"_blank" = class=3D"">osvoboda@redhat.com</a>><br class=3D""> >> >> wrote:<br class=3D""> >> >>><br class=3D""> >> >>> Hi everyone,<br class=3D""> >> >>><br class=3D""> >> >>> Even though I have nested virtualization enabled = in my Arch Linux<br class=3D""> >> >>> system<br class=3D""> >> >>> which I use to run OST, vm_run is the first test = to fail in<br class=3D""> >> >>> 004_basic_sanity<br class=3D""> >> >>> (followed by snapshots_merge and = suspend_resume_vm).<br class=3D""> >> >>><br class=3D""> >> >>> Can you point me to what I might be missing? I = believe I get the same<br class=3D""> >> >>> failure even on Fedora.<br class=3D""> >> >>><br class=3D""> >> >>> This is what host0's CPU capabilities look like = (vmx is there):<br class=3D""> >> >>> [root@lago-basic-suite-master-<wbr class=3D"">host0 = ~]# cat /proc/cpuinfo<br class=3D""> >> >>> processor : 0<br class=3D""> >> >>> vendor_id : GenuineIntel<br class=3D"">= >> >>> cpu family : 6<br class=3D""> >> >>> model : 44<br class=3D"">= >> >>> model name : Westmere = E56xx/L56xx/X56xx (Nehalem-C)<br class=3D""> >> >>> stepping : 1<br class=3D""> >> >>> microcode : 0x1<br class=3D""> >> >>> cpu MHz : 2711.988<br = class=3D""> >> >>> cache size : 16384 KB<br class=3D""> >> >>> physical id : 0<br class=3D""> >> >>> siblings : 1<br class=3D""> >> >>> core id : 0<br = class=3D""> >> >>> cpu cores : 1<br class=3D""> >> >>> apicid : 0<br class=3D"">= >> >>> initial apicid : 0<br class=3D""> >> >>> fpu : yes<br class=3D""> >> >>> fpu_exception : yes<br class=3D""> >> >>> cpuid level : 11<br class=3D""> >> >>> wp : yes<br class=3D""> >> >>> flags : fpu vme de pse = tsc msr pae mce cx8 apic sep mtrr pge<br class=3D""> >> >>> mca<br class=3D""> >> >>> cmov pat pse36 clflush mmx fxsr sse sse2 syscall = nx lm constant_tsc<br class=3D""> >> >>> rep_good<br class=3D""> >> >>> nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 = sse4_2 x2apic<br class=3D""> >> >>> popcnt aes<br class=3D""> >> >>> hypervisor lahf_lm arat tpr_shadow vnmi = flexpriority ept vpid<br class=3D""> >> >>> bogomips : 5423.97<br class=3D""> >> >>> clflush size : 64<br class=3D""> >> >>> cache_alignment : 64<br class=3D""> >> >>> address sizes : 40 bits physical, 48 = bits virtual<br class=3D""> >> >>> power management:<br class=3D""> >> >>><br class=3D""> >> >>> journalctl -b on host0 shows that libvirt = complains about NUMA<br class=3D""> >> >>> configuration:<br class=3D""> >> >>><br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]: libvirt<br class=3D""> >> >>> version: 2.0.0, package: 10.el7_3.4 (CentOS = BuildSystem<br class=3D""> >> >>> <<a href=3D"http://bugs.centos.org/" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://bugs.centos.org</a>>;, <a href=3D"tel:2017-01-17-23" = value=3D"+12017011723" target=3D"_blank" = class=3D"">2017-01-17-23</a>:37:48, <a = href=3D"http://c1bm.rdu2.centos.org/" rel=3D"noreferrer" target=3D"_blank"= class=3D"">c1bm.rdu2.centos.org</a>)<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kernel: ovirtmgmt: port<br class=3D""> >> >>> 2(vnet0) entered disabled state<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kernel: device vnet0<br class=3D""> >> >>> left<br class=3D""> >> >>> promiscuous mode<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kernel: ovirtmgmt: port<br class=3D""> >> >>> 2(vnet0) entered disabled state<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]:<br class=3D""> >> >>> hostname:<br class=3D""> >> >>> lago-basic-suite-master-host0.<wbr = class=3D"">lago.local<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]: Unable<br class=3D""> >> >>> to<br class=3D""> >> >>> read from monitor: Connection reset by peer<br = class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]:<br class=3D""> >> >>> internal<br class=3D""> >> >>> error: qemu unexpectedly closed the monitor:<br = class=3D""> >> >>> 2017-02-07T11:33:23.058571Z<br class=3D""> >> >>> qemu-kvm: warning: CPU(s) not present in any NUMA = nodes: 1 2 3 4 5 6 7<br class=3D""> >> >>> 8 9<br class=3D""> >> >>> 10 11 12 13 14 15<br class=3D""> >> >>><br class=3D""> >> >>> 2017-02-07T11:33:23.058826Z qemu-kvm: warning: All = CPU(s) up to<br class=3D""> >> >>> maxcpus<br class=3D""> >> >>> should be described in NUMA config<br class=3D""> >> >>><br class=3D""> >> >>> qemu-kvm:<br class=3D""> >> >>> /builddir/build/BUILD/qemu-2.6<wbr = class=3D"">.0/target-i386/kvm.c:1736: kvm_put_msrs:<br class=3D""> >> >>> Assertion `ret =3D=3D n' failed.<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = NetworkManager[657]:<br class=3D""> >> >>> <info><br class=3D""> >> >>> [1486467203.1025] device (vnet0): state change: = disconnected -><br class=3D""> >> >>> unmanaged<br class=3D""> >> >>> (reason 'unmanaged') [30 10 3]<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kvm[22059]: 0 guests now<br class=3D""> >> >>> active<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = systemd-machined[22044]:<br class=3D""> >> >>> Machine qemu-1-vm0 terminated.<br class=3D""> >> >>><br class=3D""> >> >>> Thanks,<br class=3D""> >> >>> Ondra<br class=3D""> >> >>><br class=3D""> >> >>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >> >>> Devel mailing list<br class=3D""> >> >>> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank"= class=3D"">Devel@ovirt.org</a><br class=3D""> >> >>> <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> >> >><br class=3D""> >> >><br class=3D""> >> >><br class=3D""> >> >><br class=3D""> >> >> --<br class=3D""> >> >> Eyal Edri<br class=3D""> >> >> Associate Manager<br class=3D""> >> >> RHV DevOps<br class=3D""> >> >> EMEA ENG Virtualization R&D<br class=3D""> >> >> Red Hat Israel<br class=3D""> >> >><br class=3D""> >> >> phone: <a href=3D"tel:%2B972-9-7692018" = value=3D"+97297692018" target=3D"_blank" class=3D"">+972-9-7692018</a><br = class=3D""> >> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br = class=3D""> >> ><br class=3D""> >> ><br class=3D""> >> ><br class=3D""> >> > ______________________________<wbr = class=3D"">_________________<br class=3D""> >> > Devel mailing list<br class=3D""> >> > <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> >> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> ><br class=3D""> ><br class=3D""> </div></div></blockquote></div><br class=3D""></div> </div></blockquote><blockquote type=3D"cite" class=3D""><div = class=3D""><span class=3D"">______________________________<wbr = class=3D"">_________________</span><br class=3D""><span class=3D"">Devel = mailing list</span><br class=3D""><span class=3D""><a = href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a></span><br class=3D""><span class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a></span></div></blockquote></div></div=
</div> <br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br= class=3D""></div> </div></blockquote></div><br class=3D""></body></html>=
--Apple-Mail=_06533245-DCE0-42FC-8274-6308A4089EF3--
--Apple-Mail=_9DD013C5-B5FF-479B-9691-CD4D3FE5FA50 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8
On 10 Feb 2017, at 10:26, Michal Skrivanek <mskrivan@redhat.com> = wrote: =20 =20
On 9 Feb 2017, at 16:16, Ondrej Svoboda <osvoboda@redhat.com = <mailto:osvoboda@redhat.com>> wrote: =20 Do you mean https://github.com/lago-project/lago/pull/398 = <https://github.com/lago-project/lago/pull/398> which has been merged = for over a month? =20 The second sentence in the PR (below) is contradicted by newer, = non-recognized CPUs, such as Skylake. =20 How/why? Westmere should have been selected in that case
=20
=20 "This patch fixes the problems by selecting a minimum reasonable CPU = model for the given hardware platform. Westmere is selected unless older = or non-Intel hardware is used." =20 On Thu, Feb 9, 2017 at 4:07 PM, Michal Skrivanek <mskrivan@redhat.com = <mailto:mskrivan@redhat.com>> wrote: What happened to Milan' PR from a while ago addressing this exact = situation? =20 On 08 Feb 2017, at 16:04, Ondrej Svoboda <osvoboda@redhat.com = <mailto:osvoboda@redhat.com>> wrote: =20
In my case, simply adding Skylake-Client a supported CPU family did =
=20 i wonder if Westmere is a good fallback -- it works for you on = Broadwell, right? =20 On Wed, Feb 8, 2017 at 1:58 PM, Nadav Goldin <ngoldin@redhat.com = <mailto:ngoldin@redhat.com>> wrote: I would first try testing it without OST, because in OST it will =
the CPU via the cluster family(which is controlled in virt.py). You can try specifying the 'cpu_model' in the init file, skipping the = 'cpu family' logic, something like: =20
cat LagoInitFile domains: vm-el73: memory: 2048 service_provider: systemd cpu_model: Broadwell nics: - net: lago disks: - template_name: el7.3-base type: template name: root dev: vda format: qcow2 nets: lago: type: nat dhcp: start: 100 end: 254 management: true dns_domain_name: lago.local =20 lago init && lago start =20 Then install lago again in the VM, copy the same init file, and = check if for different combinations of cpu_model it works for you - would give us a hint how to solve this. The 'cpu_model' basically =
to this xml definition in libvirt: <cpu mode=3D'custom' match=3D'exact'> <model fallback=3D'allow'>Broadwell</model> <topology sockets=3D'2' cores=3D'1' threads=3D'1'/> <feature policy=3D'optional' name=3D'vmx'/> <feature policy=3D'optional' name=3D'svm'/> </cpu> =20 I tried manually editing it also to host-passthrough, but still = failed on the same error. The thing is that the 'kvm_put_msrs: Assertion = `ret =3D=3D n' failed.' error doesn't give any indication where it = failed(or if the cpu is missing a flag), maybe there is a way to debug this at qemu/kvm level? I'm not sure. =20 =20 =20 =20 =20 =20 On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <osvoboda@redhat.com = <mailto:osvoboda@redhat.com>> wrote:
It is a Skylake-H, and I can see it is not mentioned in = lago/virt.py.
I guess I'll step through the code (as well as other places = discovered by 'git grep cpu') and see if I could solve this by adding the = Skylake family to _CPU_FAMILIES.
Do you have other pointers?
Thanks, Ondra
On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <ngoldin@redhat.com = <mailto:ngoldin@redhat.com>> wrote:
What is the host CPU you are using? I came across the same error few days ago, but without running =
OST, I
tried running with Lago: fc24 host -> el7 vm -> el7 vm.
I have a slight suspect that it is related to the CPU model we configure in libvirt, I tried a mixture of few combinations(host-pass-through, pinning down the CPU model), but = it always failed on the same error: kvm_put_msrs: Assertion `ret =3D=3D n' failed.
My CPU is Broadwell btw.
Milan, any ideas? you think it might be related?
Nadav.
On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda = <osvoboda@redhat.com <mailto:osvoboda@redhat.com>> wrote:
Yes, I stated that in my message.
root@osvoboda-t460p /home/src/ovirt-system-tests (git)-[master] = # cat /sys/module/kvm_intel/parameters/nested :( Y
On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <eedri@redhat.com = <mailto:eedri@redhat.com>> wrote: > > Did you follow the instructions on [1] ? > > Specifically, verifying ' cat = /sys/module/kvm_intel/parameters/nested > ' > gives you 'Y'. > > [1] > > = http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installati= on.html = <http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/installat= ion.html> > > On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda = <osvoboda@redhat.com <mailto:osvoboda@redhat.com>> > wrote: >> >> Hi everyone, >> >> Even though I have nested virtualization enabled in my Arch = Linux >> system >> which I use to run OST, vm_run is the first test to fail in >> 004_basic_sanity >> (followed by snapshots_merge and suspend_resume_vm). >> >> Can you point me to what I might be missing? I believe I get =
>> failure even on Fedora. >> >> This is what host0's CPU capabilities look like (vmx is =
>> [root@lago-basic-suite-master-host0 ~]# cat /proc/cpuinfo >> processor : 0 >> vendor_id : GenuineIntel >> cpu family : 6 >> model : 44 >> model name : Westmere E56xx/L56xx/X56xx (Nehalem-C) >> stepping : 1 >> microcode : 0x1 >> cpu MHz : 2711.988 >> cache size : 16384 KB >> physical id : 0 >> siblings : 1 >> core id : 0 >> cpu cores : 1 >> apicid : 0 >> initial apicid : 0 >> fpu : yes >> fpu_exception : yes >> cpuid level : 11 >> wp : yes >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep = mtrr pge >> mca >> cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm = constant_tsc >> rep_good >> nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 = x2apic >> popcnt aes >> hypervisor lahf_lm arat tpr_shadow vnmi flexpriority ept vpid >> bogomips : 5423.97 >> clflush size : 64 >> cache_alignment : 64 >> address sizes : 40 bits physical, 48 bits virtual >> power management: >> >> journalctl -b on host0 shows that libvirt complains about = NUMA >> configuration: >> >> Feb 07 06:33:23 lago-basic-suite-master-host0 =
>> version: 2.0.0, package: 10.el7_3.4 (CentOS BuildSystem >> <http://bugs.centos.org <http://bugs.centos.org/>>, = 2017-01-17-23 <tel:2017-01-17-23>:37:48, c1bm.rdu2.centos.org = <http://c1bm.rdu2.centos.org/>) >> Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: = ovirtmgmt: port >> 2(vnet0) entered disabled state >> Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: device = vnet0 >> left >> promiscuous mode >> Feb 07 06:33:23 lago-basic-suite-master-host0 kernel: = ovirtmgmt: port >> 2(vnet0) entered disabled state >> Feb 07 06:33:23 lago-basic-suite-master-host0 =
>> hostname: >> lago-basic-suite-master-host0.lago.local >> Feb 07 06:33:23 lago-basic-suite-master-host0 =
>> to >> read from monitor: Connection reset by peer >> Feb 07 06:33:23 lago-basic-suite-master-host0 =
And if it was and didn=E2=80=99t work for you then it is a nested = virtualization compatibility bug you should report to QEMU/KVM folks. the trick: https://github.com/lago-project/lago/pull/448 = <https://github.com/lago-project/lago/pull/448> pick translates the same there): libvirtd[12888]: libvirt libvirtd[12888]: libvirtd[12888]: Unable libvirtd[12888]:
>> internal >> error: qemu unexpectedly closed the monitor: >> 2017-02-07T11:33:23.058571Z >> qemu-kvm: warning: CPU(s) not present in any NUMA nodes: 1 2 = 3 4 5 6 7 >> 8 9 >> 10 11 12 13 14 15 >> >> 2017-02-07T11:33:23.058826Z qemu-kvm: warning: All CPU(s) up = to >> maxcpus >> should be described in NUMA config >> >> qemu-kvm: >> /builddir/build/BUILD/qemu-2.6.0/target-i386/kvm.c:1736: = kvm_put_msrs: >> Assertion `ret =3D=3D n' failed. >> Feb 07 06:33:23 lago-basic-suite-master-host0 = NetworkManager[657]: >> <info> >> [1486467203.1025] device (vnet0): state change: disconnected = -> >> unmanaged >> (reason 'unmanaged') [30 10 3] >> Feb 07 06:33:23 lago-basic-suite-master-host0 kvm[22059]: 0 = guests now >> active >> Feb 07 06:33:23 lago-basic-suite-master-host0 = systemd-machined[22044]: >> Machine qemu-1-vm0 terminated. >> >> Thanks, >> Ondra >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org <mailto:Devel@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> > > > > > -- > Eyal Edri > Associate Manager > RHV DevOps > EMEA ENG Virtualization R&D > Red Hat Israel > > phone: +972-9-7692018 <tel:%2B972-9-7692018> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
=20 _______________________________________________ Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel>
Devel mailing list Devel@ovirt.org <mailto:Devel@ovirt.org> http://lists.ovirt.org/mailman/listinfo/devel = <http://lists.ovirt.org/mailman/listinfo/devel> =20 =20
--Apple-Mail=_9DD013C5-B5FF-479B-9691-CD4D3FE5FA50 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 10 Feb 2017, at 10:26, Michal Skrivanek <<a = href=3D"mailto:mskrivan@redhat.com" class=3D"">mskrivan@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><meta= http-equiv=3D"Content-Type" content=3D"text/html charset=3Dus-ascii" = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D""><br = class=3D""><div class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 9 Feb 2017, at 16:16, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" class=3D"">osvoboda@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">Do you mean <a = href=3D"https://github.com/lago-project/lago/pull/398" = class=3D"">https://github.com/lago-project/lago/pull/398</a> which has = been merged for over a month?<br class=3D""><br class=3D""></div>The = second sentence in the PR (below) is contradicted by newer, = non-recognized CPUs, such as Skylake.<br = class=3D""></div></div></blockquote><div class=3D""><br = class=3D""></div>How/why? Westmere should have been selected in that = case</div></div></div></blockquote><div><br class=3D""></div>And if it = was and didn=E2=80=99t work for you then it is a nested virtualization = compatibility bug you should report to QEMU/KVM folks.</div><div><br = class=3D""></div><div><blockquote type=3D"cite" class=3D""><div = class=3D""><div style=3D"word-wrap: break-word; -webkit-nbsp-mode: = space; -webkit-line-break: after-white-space;" class=3D""><div = class=3D""><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div class=3D""><div = class=3D""><div class=3D""><br class=3D"">"This patch fixes the problems = by selecting a minimum reasonable CPU model for the given hardware = platform. Westmere is selected unless older or non-Intel hardware is = used."</div></div></div></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Thu, Feb 9, 2017 at 4:07 PM, = Michal Skrivanek <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:mskrivan@redhat.com" target=3D"_blank" = class=3D"">mskrivan@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"auto" = class=3D""><div class=3D""></div><div class=3D"">What happened to Milan' = PR from a while ago addressing this exact situation?</div><div = class=3D""><div class=3D"h5"><div class=3D""><br class=3D"">On 08 Feb = 2017, at 16:04, Ondrej Svoboda <<a href=3D"mailto:osvoboda@redhat.com" = target=3D"_blank" class=3D"">osvoboda@redhat.com</a>> wrote:<br = class=3D""><br class=3D""></div><blockquote type=3D"cite" class=3D""><div = class=3D""><div dir=3D"ltr" class=3D""><div class=3D""><div class=3D"">In = my case, simply adding Skylake-Client a supported CPU family did the = trick: <a href=3D"https://github.com/lago-project/lago/pull/448" = target=3D"_blank" class=3D"">https://github.com/lago-<wbr = class=3D"">project/lago/pull/448</a><br class=3D""><br class=3D""></div>i = wonder if Westmere is a good fallback -- it works for you on Broadwell, = right?<br class=3D""></div></div><div class=3D"gmail_extra"><br = class=3D""><div class=3D"gmail_quote">On Wed, Feb 8, 2017 at 1:58 PM, = Nadav Goldin <span dir=3D"ltr" class=3D""><<a = href=3D"mailto:ngoldin@redhat.com" target=3D"_blank" = class=3D"">ngoldin@redhat.com</a>></span> wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 = .8ex;border-left:1px #ccc solid;padding-left:1ex">I would first try = testing it without OST, because in OST it will pick<br class=3D""> the CPU via the cluster family(which is controlled in virt.py). You<br = class=3D""> can try specifying the 'cpu_model' in the init file, skipping the = 'cpu<br class=3D""> family' logic, something like:<br class=3D""> <br class=3D""> > cat LagoInitFile<br class=3D""> domains:<br class=3D""> vm-el73:<br class=3D""> memory: 2048<br class=3D""> service_provider: systemd<br class=3D""> cpu_model: Broadwell<br class=3D""> nics:<br class=3D""> - net: lago<br class=3D""> disks:<br class=3D""> - template_name: el7.3-base<br class=3D""> type: template<br class=3D""> name: root<br class=3D""> dev: vda<br class=3D""> format: qcow2<br class=3D""> nets:<br class=3D""> lago:<br class=3D""> type: nat<br class=3D""> dhcp:<br class=3D""> start: 100<br class=3D""> end: 254<br class=3D""> management: true<br class=3D""> dns_domain_name: lago.local<br class=3D""> <br class=3D""> > lago init && lago start<br class=3D""> <br class=3D""> Then install lago again in the VM, copy the same init file, and check<br = class=3D""> if for different combinations of cpu_model it works for you - would<br = class=3D""> give us a hint how to solve this. The 'cpu_model' basically = translates<br class=3D""> to this xml definition in libvirt:<br class=3D""> <cpu mode=3D'custom' match=3D'exact'><br class=3D""> <model fallback=3D'allow'>Broadwell</mo<wbr = class=3D"">del><br class=3D""> <topology sockets=3D'2' cores=3D'1' threads=3D'1'/><br= class=3D""> <feature policy=3D'optional' name=3D'vmx'/><br = class=3D""> <feature policy=3D'optional' name=3D'svm'/><br = class=3D""> </cpu><br class=3D""> <br class=3D""> I tried manually editing it also to host-passthrough, but still = failed<br class=3D""> on the same error. The thing is that the 'kvm_put_msrs: Assertion = `ret<br class=3D""> =3D=3D n' failed.' error doesn't give any indication where it failed(or = if<br class=3D""> the cpu is missing a flag), maybe there is a way to debug this at<br = class=3D""> qemu/kvm level? I'm not sure.<br class=3D""> <div class=3D"m_-909161191278510801HOEnZb"><div = class=3D"m_-909161191278510801h5"><br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> <br class=3D""> On Wed, Feb 8, 2017 at 1:18 PM, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" target=3D"_blank" = class=3D"">osvoboda@redhat.com</a>> wrote:<br class=3D""> > It is a Skylake-H, and I can see it is not mentioned in = lago/virt.py.<br class=3D""> ><br class=3D""> > I guess I'll step through the code (as well as other places = discovered by<br class=3D""> > 'git grep cpu') and see if I could solve this by adding the Skylake = family<br class=3D""> > to _CPU_FAMILIES.<br class=3D""> ><br class=3D""> > Do you have other pointers?<br class=3D""> ><br class=3D""> > Thanks,<br class=3D""> > Ondra<br class=3D""> ><br class=3D""> > On Tue, Feb 7, 2017 at 10:40 PM, Nadav Goldin <<a = href=3D"mailto:ngoldin@redhat.com" target=3D"_blank" = class=3D"">ngoldin@redhat.com</a>> wrote:<br class=3D""> >><br class=3D""> >> What is the host CPU you are using?<br class=3D""> >> I came across the same error few days ago, but without running = OST, I<br class=3D""> >> tried running with Lago:<br class=3D""> >> fc24 host -> el7 vm -> el7 vm.<br class=3D""> >><br class=3D""> >> I have a slight suspect that it is related to the CPU model = we<br class=3D""> >> configure in libvirt, I tried a mixture of few<br class=3D""> >> combinations(host-pass-through<wbr class=3D"">, pinning down = the CPU model), but it<br class=3D""> >> always failed on the same error:<br class=3D""> >> kvm_put_msrs: Assertion `ret =3D=3D n' failed.<br class=3D""> >><br class=3D""> >> My CPU is Broadwell btw.<br class=3D""> >><br class=3D""> >><br class=3D""> >> Milan, any ideas? you think it might be related?<br class=3D""> >><br class=3D""> >> Nadav.<br class=3D""> >><br class=3D""> >><br class=3D""> >><br class=3D""> >> On Tue, Feb 7, 2017 at 11:14 PM, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" target=3D"_blank" = class=3D"">osvoboda@redhat.com</a>><br class=3D""> >> wrote:<br class=3D""> >> > Yes, I stated that in my message.<br class=3D""> >> ><br class=3D""> >> > root@osvoboda-t460p /home/src/ovirt-system-tests = (git)-[master] # cat<br class=3D""> >> > /sys/module/kvm_intel/paramete<wbr class=3D"">rs/nested<br = class=3D""> >> > :(<br class=3D""> >> > Y<br class=3D""> >> ><br class=3D""> >> > On Tue, Feb 7, 2017 at 1:39 PM, Eyal Edri <<a = href=3D"mailto:eedri@redhat.com" target=3D"_blank" = class=3D"">eedri@redhat.com</a>> wrote:<br class=3D""> >> >><br class=3D""> >> >> Did you follow the instructions on [1] ?<br class=3D""> >> >><br class=3D""> >> >> Specifically, verifying ' cat = /sys/module/kvm_intel/paramete<wbr class=3D"">rs/nested<br class=3D""> >> >> '<br class=3D""> >> >> gives you 'Y'.<br class=3D""> >> >><br class=3D""> >> >> [1]<br class=3D""> >> >><br class=3D""> >> >> <a = href=3D"http://ovirt-system-tests.readthedocs.io/en/latest/docs/general/in= stallation.html" rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://ovirt-system-tests.read<wbr = class=3D"">thedocs.io/en/latest/docs/gene<wbr = class=3D"">ral/installation.html</a><br class=3D""> >> >><br class=3D""> >> >> On Tue, Feb 7, 2017 at 2:29 PM, Ondrej Svoboda <<a = href=3D"mailto:osvoboda@redhat.com" target=3D"_blank" = class=3D"">osvoboda@redhat.com</a>><br class=3D""> >> >> wrote:<br class=3D""> >> >>><br class=3D""> >> >>> Hi everyone,<br class=3D""> >> >>><br class=3D""> >> >>> Even though I have nested virtualization enabled = in my Arch Linux<br class=3D""> >> >>> system<br class=3D""> >> >>> which I use to run OST, vm_run is the first test = to fail in<br class=3D""> >> >>> 004_basic_sanity<br class=3D""> >> >>> (followed by snapshots_merge and = suspend_resume_vm).<br class=3D""> >> >>><br class=3D""> >> >>> Can you point me to what I might be missing? I = believe I get the same<br class=3D""> >> >>> failure even on Fedora.<br class=3D""> >> >>><br class=3D""> >> >>> This is what host0's CPU capabilities look like = (vmx is there):<br class=3D""> >> >>> [root@lago-basic-suite-master-<wbr class=3D"">host0 = ~]# cat /proc/cpuinfo<br class=3D""> >> >>> processor : 0<br class=3D""> >> >>> vendor_id : GenuineIntel<br class=3D"">= >> >>> cpu family : 6<br class=3D""> >> >>> model : 44<br class=3D"">= >> >>> model name : Westmere = E56xx/L56xx/X56xx (Nehalem-C)<br class=3D""> >> >>> stepping : 1<br class=3D""> >> >>> microcode : 0x1<br class=3D""> >> >>> cpu MHz : 2711.988<br = class=3D""> >> >>> cache size : 16384 KB<br class=3D""> >> >>> physical id : 0<br class=3D""> >> >>> siblings : 1<br class=3D""> >> >>> core id : 0<br = class=3D""> >> >>> cpu cores : 1<br class=3D""> >> >>> apicid : 0<br class=3D"">= >> >>> initial apicid : 0<br class=3D""> >> >>> fpu : yes<br class=3D""> >> >>> fpu_exception : yes<br class=3D""> >> >>> cpuid level : 11<br class=3D""> >> >>> wp : yes<br class=3D""> >> >>> flags : fpu vme de pse = tsc msr pae mce cx8 apic sep mtrr pge<br class=3D""> >> >>> mca<br class=3D""> >> >>> cmov pat pse36 clflush mmx fxsr sse sse2 syscall = nx lm constant_tsc<br class=3D""> >> >>> rep_good<br class=3D""> >> >>> nopl xtopology pni pclmulqdq vmx ssse3 cx16 sse4_1 = sse4_2 x2apic<br class=3D""> >> >>> popcnt aes<br class=3D""> >> >>> hypervisor lahf_lm arat tpr_shadow vnmi = flexpriority ept vpid<br class=3D""> >> >>> bogomips : 5423.97<br class=3D""> >> >>> clflush size : 64<br class=3D""> >> >>> cache_alignment : 64<br class=3D""> >> >>> address sizes : 40 bits physical, 48 = bits virtual<br class=3D""> >> >>> power management:<br class=3D""> >> >>><br class=3D""> >> >>> journalctl -b on host0 shows that libvirt = complains about NUMA<br class=3D""> >> >>> configuration:<br class=3D""> >> >>><br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]: libvirt<br class=3D""> >> >>> version: 2.0.0, package: 10.el7_3.4 (CentOS = BuildSystem<br class=3D""> >> >>> <<a href=3D"http://bugs.centos.org/" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://bugs.centos.org</a>>;, <a href=3D"tel:2017-01-17-23" = value=3D"+12017011723" target=3D"_blank" = class=3D"">2017-01-17-23</a>:37:48, <a = href=3D"http://c1bm.rdu2.centos.org/" rel=3D"noreferrer" target=3D"_blank"= class=3D"">c1bm.rdu2.centos.org</a>)<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kernel: ovirtmgmt: port<br class=3D""> >> >>> 2(vnet0) entered disabled state<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kernel: device vnet0<br class=3D""> >> >>> left<br class=3D""> >> >>> promiscuous mode<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kernel: ovirtmgmt: port<br class=3D""> >> >>> 2(vnet0) entered disabled state<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]:<br class=3D""> >> >>> hostname:<br class=3D""> >> >>> lago-basic-suite-master-host0.<wbr = class=3D"">lago.local<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]: Unable<br class=3D""> >> >>> to<br class=3D""> >> >>> read from monitor: Connection reset by peer<br = class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = libvirtd[12888]:<br class=3D""> >> >>> internal<br class=3D""> >> >>> error: qemu unexpectedly closed the monitor:<br = class=3D""> >> >>> 2017-02-07T11:33:23.058571Z<br class=3D""> >> >>> qemu-kvm: warning: CPU(s) not present in any NUMA = nodes: 1 2 3 4 5 6 7<br class=3D""> >> >>> 8 9<br class=3D""> >> >>> 10 11 12 13 14 15<br class=3D""> >> >>><br class=3D""> >> >>> 2017-02-07T11:33:23.058826Z qemu-kvm: warning: All = CPU(s) up to<br class=3D""> >> >>> maxcpus<br class=3D""> >> >>> should be described in NUMA config<br class=3D""> >> >>><br class=3D""> >> >>> qemu-kvm:<br class=3D""> >> >>> /builddir/build/BUILD/qemu-2.6<wbr = class=3D"">.0/target-i386/kvm.c:1736: kvm_put_msrs:<br class=3D""> >> >>> Assertion `ret =3D=3D n' failed.<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = NetworkManager[657]:<br class=3D""> >> >>> <info><br class=3D""> >> >>> [1486467203.1025] device (vnet0): state change: = disconnected -><br class=3D""> >> >>> unmanaged<br class=3D""> >> >>> (reason 'unmanaged') [30 10 3]<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = kvm[22059]: 0 guests now<br class=3D""> >> >>> active<br class=3D""> >> >>> Feb 07 06:33:23 lago-basic-suite-master-host0 = systemd-machined[22044]:<br class=3D""> >> >>> Machine qemu-1-vm0 terminated.<br class=3D""> >> >>><br class=3D""> >> >>> Thanks,<br class=3D""> >> >>> Ondra<br class=3D""> >> >>><br class=3D""> >> >>> ______________________________<wbr = class=3D"">_________________<br class=3D""> >> >>> Devel mailing list<br class=3D""> >> >>> <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank"= class=3D"">Devel@ovirt.org</a><br class=3D""> >> >>> <a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> >> >><br class=3D""> >> >><br class=3D""> >> >><br class=3D""> >> >><br class=3D""> >> >> --<br class=3D""> >> >> Eyal Edri<br class=3D""> >> >> Associate Manager<br class=3D""> >> >> RHV DevOps<br class=3D""> >> >> EMEA ENG Virtualization R&D<br class=3D""> >> >> Red Hat Israel<br class=3D""> >> >><br class=3D""> >> >> phone: <a href=3D"tel:%2B972-9-7692018" = value=3D"+97297692018" target=3D"_blank" class=3D"">+972-9-7692018</a><br = class=3D""> >> >> irc: eedri (on #tlv #rhev-dev #rhev-integ)<br = class=3D""> >> ><br class=3D""> >> ><br class=3D""> >> ><br class=3D""> >> > ______________________________<wbr = class=3D"">_________________<br class=3D""> >> > Devel mailing list<br class=3D""> >> > <a href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a><br class=3D""> >> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/mailman<wbr = class=3D"">/listinfo/devel</a><br class=3D""> ><br class=3D""> ><br class=3D""> </div></div></blockquote></div><br class=3D""></div> </div></blockquote><blockquote type=3D"cite" class=3D""><div = class=3D""><span class=3D"">______________________________<wbr = class=3D"">_________________</span><br class=3D""><span class=3D"">Devel = mailing list</span><br class=3D""><span class=3D""><a = href=3D"mailto:Devel@ovirt.org" target=3D"_blank" = class=3D"">Devel@ovirt.org</a></span><br class=3D""><span class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/devel" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a></span></div></blockquote></div></div=
</div> <br class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Devel mailing list<br class=3D""> <a href=3D"mailto:Devel@ovirt.org" class=3D"">Devel@ovirt.org</a><br = class=3D""> <a href=3D"http://lists.ovirt.org/mailman/listinfo/devel" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/devel</a><br class=3D""></blockquote></div><br= class=3D""></div> </div></blockquote></div><br class=3D""></div></div></blockquote></div><br= class=3D""></body></html>=
--Apple-Mail=_9DD013C5-B5FF-479B-9691-CD4D3FE5FA50--
participants (4)
-
Eyal Edri -
Michal Skrivanek -
Nadav Goldin -
Ondrej Svoboda