Low Performance (KVM Vs VMware Hypervisor) When running multi-process application
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup: 1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU). The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). *The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.* Any thoughts on the below observations? Any suggestions? - KVM Guest VMs degraded performance when running multi-process applications. - High FUTEX time (Seen on the Guest VM when passing traffic). - *High SY: *System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.) -Rav Ya
what kind of CPUs are you using? Rav Ya <ravin.ya90@gmail.com> schrieb am Di., 15. Sept. 2020, 16:58:
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup:
1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU).
The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). *The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.*
Any thoughts on the below observations? Any suggestions?
- KVM Guest VMs degraded performance when running multi-process applications. - High FUTEX time (Seen on the Guest VM when passing traffic). - *High SY: *System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.)
-Rav Ya
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QSEUE5VM4UCRT7...
Hi Arman, Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz *The VM is configured for host CPU pass through and pinned to 6 CPUs.* Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 6 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 2593.906 BogoMIPS: 5187.81 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-5 Thank You -RY On Tue, Sep 15, 2020 at 6:21 PM Arman Khalatyan <arm2arm@gmail.com> wrote:
what kind of CPUs are you using?
Rav Ya <ravin.ya90@gmail.com> schrieb am Di., 15. Sept. 2020, 16:58:
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup:
1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU).
The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). *The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.*
Any thoughts on the below observations? Any suggestions?
- KVM Guest VMs degraded performance when running multi-process applications. - High FUTEX time (Seen on the Guest VM when passing traffic). - *High SY: *System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.)
-Rav Ya
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QSEUE5VM4UCRT7...
What about the Vmware ESX host - does it have the same CPU ? Best Regards, Strahil Nikolov В сряда, 16 септември 2020 г., 01:58:30 Гринуич+3, Rav Ya <ravin.ya90@gmail.com> написа: Hi Arman, Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz The VM is configured for host CPU pass through and pinned to 6 CPUs. Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 6 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 2593.906 BogoMIPS: 5187.81 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-5 Thank You -RY On Tue, Sep 15, 2020 at 6:21 PM Arman Khalatyan <arm2arm@gmail.com> wrote:
what kind of CPUs are you using?
Rav Ya <ravin.ya90@gmail.com> schrieb am Di., 15. Sept. 2020, 16:58:
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup: 1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU). The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.
Any thoughts on the below observations? Any suggestions?
* KVM Guest VMs degraded performance when running multi-process applications. * High FUTEX time (Seen on the Guest VM when passing traffic). * High SY: System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.)
-Rav Ya
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QSEUE5VM4UCRT7...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HDX5I45ALOPTWK...
ok will try on our env with passthrough, could you please send how you passthrough the cpu? simply over the ovirt gui? Rav Ya <ravin.ya90@gmail.com> schrieb am Mi., 16. Sept. 2020, 00:56:
Hi Arman,
Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz
*The VM is configured for host CPU pass through and pinned to 6 CPUs.*
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 6 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 2593.906 BogoMIPS: 5187.81 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-5
Thank You -RY
On Tue, Sep 15, 2020 at 6:21 PM Arman Khalatyan <arm2arm@gmail.com> wrote:
what kind of CPUs are you using?
Rav Ya <ravin.ya90@gmail.com> schrieb am Di., 15. Sept. 2020, 16:58:
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup:
1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU).
The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). *The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.*
Any thoughts on the below observations? Any suggestions?
- KVM Guest VMs degraded performance when running multi-process applications. - High FUTEX time (Seen on the Guest VM when passing traffic). - *High SY: *System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.)
-Rav Ya
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QSEUE5VM4UCRT7...
In the VM 'edit' settings you can pick the 'Host' tab on the left, specify 'Specific Host(s)' , define the migration mode (I'm using both Auto and Manual as my cluster has the same CPU type) and last enable 'Pass-Through Host CPU' and save the VM. Then you can power it up and it should be good to go. Best Regards, Strahil Nikolov В сряда, 16 септември 2020 г., 17:29:10 Гринуич+3, Arman Khalatyan <arm2arm@gmail.com> написа: ok will try on our env with passthrough, could you please send how you passthrough the cpu? simply over the ovirt gui? Rav Ya <ravin.ya90@gmail.com> schrieb am Mi., 16. Sept. 2020, 00:56:
Hi Arman,
Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz
The VM is configured for host CPU pass through and pinned to 6 CPUs.
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 6 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 2593.906 BogoMIPS: 5187.81 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-5
Thank You -RY
On Tue, Sep 15, 2020 at 6:21 PM Arman Khalatyan <arm2arm@gmail.com> wrote:
what kind of CPUs are you using?
Rav Ya <ravin.ya90@gmail.com> schrieb am Di., 15. Sept. 2020, 16:58:
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup: 1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU). The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.
Any thoughts on the below observations? Any suggestions?
* KVM Guest VMs degraded performance when running multi-process applications. * High FUTEX time (Seen on the Guest VM when passing traffic). * High SY: System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.)
-Rav Ya
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QSEUE5VM4UCRT7...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2435WWIOIGERH2...
Hello Everyone, The mystery regarding the performance (VMware Vs KVM) is uncovered. Seems like the VMware is not pinning the CPUs properly and using extra MHz CPU to go faster. If I set an upper limit on the MHz usage on the VMware. I get the same performance as KVM. Thanks a lot for your time. Thank you, Rav ya On Wed, Sep 16, 2020 at 10:27 AM Arman Khalatyan <arm2arm@gmail.com> wrote:
ok will try on our env with passthrough, could you please send how you passthrough the cpu? simply over the ovirt gui?
Rav Ya <ravin.ya90@gmail.com> schrieb am Mi., 16. Sept. 2020, 00:56:
Hi Arman,
Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz
*The VM is configured for host CPU pass through and pinned to 6 CPUs.*
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 6 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz Stepping: 4 CPU MHz: 2593.906 BogoMIPS: 5187.81 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-5
Thank You -RY
On Tue, Sep 15, 2020 at 6:21 PM Arman Khalatyan <arm2arm@gmail.com> wrote:
what kind of CPUs are you using?
Rav Ya <ravin.ya90@gmail.com> schrieb am Di., 15. Sept. 2020, 16:58:
Hello Everyone, Please advice. Any help will be highly appreciated. Thank you in advance. Test Setup:
1. oVirt Centos 7.8 Virtulization Host 2. Guest VM Centos 7.8 (Mutiqueue enabled 6 vCPUs with 6 Rx Tx Queues) 3. The vCPUs are configured for host pass through (Pinned CPU).
The Guest VM runs the application in userspace. The Application consists of the parent process that reads packets in raw socket mode from the interface and forwards then to child processes (~vCPUs) via IPC (shared memory – pipes). *The performance (throughput / CPU utilization) that I get with KVM is half of what I get with VMware.*
Any thoughts on the below observations? Any suggestions?
- KVM Guest VMs degraded performance when running multi-process applications. - High FUTEX time (Seen on the Guest VM when passing traffic). - *High SY: *System CPU time spent in kernel space (Seen on both Hypervisor and the Guest VMs only when running my application.)
-Rav Ya
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QSEUE5VM4UCRT7...
participants (3)
-
Arman Khalatyan -
Rav Ya -
Strahil Nikolov