Basic setup notes: 3 node HCI running oVirt 4.3.3 using nodeNG for hosts. Storage is SSD backed with 10Gb network dedicated to gluster with jumbo frames enabled. The ovirt management network (which also acts as VM network) is 1Gb network Hosts are dell R720s w/ 256gb ram and E5-2690 procs I have a VM configured with 16Gb ram and 6 virtual CPUs. When this VM does a heavy operation (in this case it's dumping a large DB from a remote server) the load spikes quickly. When this happens the VM becomes unresponsive and in some cases I get cpu soft lock messages. I'm trying to determine where the bottleneck here is and how I can prevent the VM from becoming unresponsive when doing heavy tasks. Here is info from syslog showing the soft lockup: Jun 26 16:13:10 roble kernel: NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [pg_dump:4025] Jun 26 16:13:10 roble kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev parport_pc parport virtio_rng i2c_piix4 pcspkr ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic virtio_console virtio_net virtio_scsi pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel qxl drm_kms_helper syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops ttm drm floppy ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod Jun 26 16:13:10 roble kernel: CPU: 5 PID: 4025 Comm: pg_dump Kdump: loaded Not tainted 3.10.0-957.21.2.el7.x86_64 #1 Jun 26 16:13:10 roble kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 Jun 26 16:13:10 roble kernel: task: ffffa01443499040 ti: ffffa014b24a4000 task.ti: ffffa014b24a4000 Jun 26 16:13:10 roble kernel: RIP: 0010:[<ffffffffaeb113ea>] [<ffffffffaeb113ea>] generic_exec_single+0xfa/0x1b0 Jun 26 16:13:10 roble kernel: RSP: 0018:ffffa014b24a7c30 EFLAGS: 00000202 Jun 26 16:13:10 roble kernel: RAX: 0000000000000010 RBX: ffffa014b24a7c00 RCX: 0000000000000030 Jun 26 16:13:10 roble kernel: RDX: 000000000000ffff RSI: 0000000000000010 RDI: 0000000000000286 Jun 26 16:13:10 roble kernel: RBP: ffffa014b24a7c78 R08: ffffffffaf213640 R09: 000000018040003f Jun 26 16:13:10 roble kernel: R10: 0000000000000001 R11: fffff83b8e91b540 R12: ffffa014b24a7bc0 Jun 26 16:13:10 roble kernel: R13: 0000000000000c9b R14: ffffa01477bf94e8 R15: ffffa01545074270 Jun 26 16:13:10 roble kernel: FS: 00007f9fefe89840(0000) GS:ffffa0172f340000(0000) knlGS:0000000000000000 Jun 26 16:13:10 roble kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 26 16:13:10 roble kernel: CR2: 00007f9fef391e90 CR3: 000000014aa00000 CR4: 00000000000606e0 Jun 26 16:13:10 roble kernel: Call Trace: Jun 26 16:13:10 roble kernel: [<ffffffffaea7a4e0>] ? leave_mm+0x110/0x110 Jun 26 16:13:10 roble kernel: [<ffffffffaea7a4e0>] ? leave_mm+0x110/0x110 Jun 26 16:13:10 roble kernel: [<ffffffffaea7a4e0>] ? leave_mm+0x110/0x110 Jun 26 16:13:10 roble kernel: [<ffffffffaeb114ff>] smp_call_function_single+0x5f/0xa0 Jun 26 16:13:10 roble kernel: [<ffffffffaed75cd5>] ? cpumask_next_and+0x35/0x50 Jun 26 16:13:10 roble kernel: [<ffffffffaeb11aab>] smp_call_function_many+0x22b/0x270 Jun 26 16:13:10 roble kernel: [<ffffffffaea7a6a8>] native_flush_tlb_others+0xb8/0xc0 Jun 26 16:13:10 roble kernel: [<ffffffffaea7a718>] flush_tlb_mm_range+0x68/0x140 Jun 26 16:13:10 roble kernel: [<ffffffffaebe4687>] tlb_flush_mmu.part.76+0x37/0xe0 Jun 26 16:13:10 roble kernel: [<ffffffffaebe5f85>] tlb_finish_mmu+0x55/0x60 Jun 26 16:13:10 roble kernel: [<ffffffffaebef624>] unmap_region+0xf4/0x140 Jun 26 16:13:10 roble kernel: [<ffffffffaecfc8d3>] ? selinux_file_free_security+0x23/0x30 Jun 26 16:13:10 roble kernel: [<ffffffffaebefbe1>] ? __vma_rb_erase+0x121/0x220 Jun 26 16:13:10 roble kernel: [<ffffffffaebf1c15>] do_munmap+0x2a5/0x480 Jun 26 16:13:10 roble kernel: [<ffffffffaebf1e55>] vm_munmap+0x65/0xb0 Jun 26 16:13:10 roble kernel: [<ffffffffaebf30e2>] SyS_munmap+0x22/0x30 Jun 26 16:13:10 roble kernel: [<ffffffffaf175ddb>] system_call_fastpath+0x22/0x27 Jun 26 16:13:10 roble kernel: [<ffffffffaf175d21>] ? system_call_after_swapgs+0xae/0x146 Jun 26 16:13:10 roble kernel: Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 bc 74 af 48 89 df e8 4a b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 Jun 26 16:13:14 roble kernel: NMI watchdog: BUG: soft lockup - CPU#4 stuck for 27s! [kworker/4:3:16530] Jun 26 16:13:14 roble kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev parport_pc parport virtio_rng i2c_piix4 pcspkr ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic virtio_console virtio_net virtio_scsi pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel qxl drm_kms_helper syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops ttm drm floppy ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod Jun 26 16:13:14 roble kernel: CPU: 4 PID: 16530 Comm: kworker/4:3 Kdump: loaded Tainted: G L ------------ 3.10.0-957.21.2.el7.x86_64 #1 Jun 26 16:13:14 roble kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 Jun 26 16:13:14 roble kernel: Workqueue: events tsc_refine_calibration_work Jun 26 16:13:14 roble kernel: task: ffffa016acf74100 ti: ffffa014f76a8000 task.ti: ffffa014f76a8000 Jun 26 16:13:14 roble kernel: RIP: 0010:[<ffffffffaefb97e0>] [<ffffffffaefb97e0>] acpi_pm_read_verified+0x10/0x60 Jun 26 16:13:14 roble kernel: RSP: 0018:ffffa014f76abdb8 EFLAGS: 00000202 Jun 26 16:13:14 roble kernel: RAX: 00000000000addf6 RBX: 0000000000000086 RCX: 0000000000f16644 Jun 26 16:13:14 roble kernel: RDX: 0000000000000608 RSI: 00000000002aac00 RDI: ffffffffaf971901 Jun 26 16:13:14 roble kernel: RBP: ffffa014f76abdb8 R08: 00000000aff71901 R09: 0001427407e2e9e0 Jun 26 16:13:14 roble kernel: R10: 0001427407e2e9e0 R11: 0000000000000000 R12: 0000000000000004 Jun 26 16:13:14 roble kernel: R13: ffffa014f76abd38 R14: ffffffffaf62ea00 R15: ffffa0172f313900 Jun 26 16:13:14 roble kernel: FS: 0000000000000000(0000) GS:ffffa0172f300000(0000) knlGS:0000000000000000 Jun 26 16:13:14 roble kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jun 26 16:13:14 roble kernel: CR2: 0000000000448d70 CR3: 000000017789a000 CR4: 00000000000606e0 Jun 26 16:13:14 roble kernel: Call Trace: Jun 26 16:13:14 roble kernel: [<ffffffffaea3426d>] tsc_read_refs+0x8d/0xb0 Jun 26 16:13:14 roble kernel: [<ffffffffaea344b2>] tsc_refine_calibration_work+0x1c2/0x220 Jun 26 16:13:14 roble kernel: [<ffffffffaeab9ebf>] process_one_work+0x17f/0x440 Jun 26 16:13:14 roble kernel: [<ffffffffaeabaf56>] worker_thread+0x126/0x3c0 Jun 26 16:13:14 roble kernel: [<ffffffffaeabae30>] ? manage_workers.isra.25+0x2a0/0x2a0 Jun 26 16:13:14 roble kernel: [<ffffffffaeac1da1>] kthread+0xd1/0xe0 Jun 26 16:13:14 roble kernel: [<ffffffffaeac1cd0>] ? insert_kthread_work+0x40/0x40 Jun 26 16:13:14 roble kernel: [<ffffffffaf175c37>] ret_from_fork_nospec_begin+0x21/0x21 Jun 26 16:13:14 roble kernel: [<ffffffffaeac1cd0>] ? insert_kthread_work+0x40/0x40 Jun 26 16:13:14 roble kernel: Code: 43 ea 74 00 30 98 fb ae c7 05 81 ea 74 00 78 00 00 00 5d c3 0f 1f 80 00 00 00 00 66 66 66 66 90 8b 15 5d 74 7a 00 55 48 89 e5 ed <89> c6 81 e6 ff ff ff 00 ed 89 c1 81 e1 ff ff ff 00 ed 25 ff ff