[ovirt-users] swap + crash

[Doron Fediuck]

On 23/06/15 17:00, paf1 at email.cz wrote:
> Hello,
> we have big problem with metadata IO error and swap issue
> with:
> OS =                    RHEL - 7 - 1.1503.el7.centos.2.8
> kernel =              3.10.0 - 229.4.2.el7.x86_64
> KVM Version =  2.1.2 - 23.el7_1.3.1
> LIBVIRT =           libvirt-1.2.8-16.el7_1.3
> gluster =              glusterfs-3.7.1-1.el7       ( replica 2 on XFS )
> Phys.MEM =       128667 MB total, 23160 MB used, 105507 MB free
> swap =                 398334 MB total, 32049 MB used, 366285 MB free
> 
> Can anybody  get any idea what's wrong ??
> System swapping continualy in spite of a lot of free memory available. (
> we added large temporary swap space  )
> 
> 
> Permanent write to /var/log/messages :
> 
> Journal: End of file while reading data: Input/output error
> ... and
> Jun 23 15:46:04 1kvm1 journal: metadata not found: Requested metadata
> element is not present
> Jun 23 15:46:06 1kvm1 journal: metadata not found: Requested metadata
> element is not present
> Jun 23 15:46:09 1kvm1 kernel: swapper/16: page allocation failure:
> order:2, mode:0x104020
> Jun 23 15:46:09 1kvm1 kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted:
> G          I --------------   3.10.0-229.4.2.el7.x86_64 #1
> Jun 23 15:46:09 1kvm1 kernel: Hardware name: Supermicro
> X10SRi-F/X10SRi-F, BIOS 1.0a 08/27/2014
> Jun 23 15:46:09 1kvm1 kernel: 0000000000104020 04badf98503e3f59
> ffff881fff403a00 ffffffff81604eaa
> Jun 23 15:46:09 1kvm1 kernel: ffff881fff403a90 ffffffff8115c620
> ffff88207fffd4e8 ffff881fff403a50
> Jun 23 15:46:09 1kvm1 kernel: ffff88207ffd8000 ffff88207ffd8000
> 0000000000000002 04badf98503e3f59
> Jun 23 15:46:09 1kvm1 kernel: Call Trace:
> Jun 23 15:46:09 1kvm1 kernel: <IRQ>  [<ffffffff81604eaa>]
> dump_stack+0x19/0x1b
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8115c620>]
> warn_alloc_failed+0x110/0x180
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81160db8>]
> __alloc_pages_nodemask+0x9a8/0xb90
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8155ab15>] ?
> tcp_v4_do_rcv+0x1b5/0x470
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8119f549>]
> alloc_pages_current+0xa9/0x170
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8115b59e>] __get_free_pages+0xe/0x50
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff811aa32e>]
> kmalloc_order_trace+0x2e/0xa0
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff811acbc9>] __kmalloc+0x219/0x230
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f02ea>]
> bnx2x_frag_alloc.isra.64+0x2a/0x40 [bnx2x]
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f15b4>]
> bnx2x_alloc_rx_data.isra.71+0x54/0x1c0 [bnx2x]
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f33bd>]
> bnx2x_rx_int+0x89d/0x1910 [bnx2x]
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8101bad9>] ? sched_clock+0x9/0x10
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810acd84>] ? task_cputime+0x44/0x80
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810400c2>] ?
> x86_acpi_suspend_lowlevel+0x12/0x170
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f485a>] bnx2x_poll+0xfa/0x3c0
> [bnx2x]
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814fcfa2>] net_rx_action+0x152/0x240
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81077bf7>] __do_softirq+0xf7/0x290
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8161671c>] call_softirq+0x1c/0x30
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81015de5>] do_softirq+0x55/0x90
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81077f95>] irq_exit+0x115/0x120
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff816172b8>] do_IRQ+0x58/0xf0
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8160c4ad>]
> common_interrupt+0x6d/0x6d
> Jun 23 15:46:09 1kvm1 kernel: <EOI>  [<ffffffff814aaa32>] ?
> cpuidle_enter_state+0x52/0xc0
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814aaa28>] ?
> cpuidle_enter_state+0x48/0xc0
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814aab65>]
> cpuidle_idle_call+0xc5/0x200
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8101d21e>] arch_cpu_idle+0xe/0x30
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810c6985>]
> cpu_startup_entry+0xf5/0x290
> Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810423ca>]
> start_secondary+0x1ba/0x230
> Jun 23 15:46:16 1kvm1 journal: metadata not found: Requested metadata
> element is not present
> Jun 23 15:46:19 1kvm1 journal: metadata not found: Requested metadata
> element is not present
> 
> regs. a lot
> Paf1
> 
> 

Hi Paf1,
The metadata issue is not the real problem. It is fixed as a part of-
https://bugzilla.redhat.com/show_bug.cgi?id=1181157

The engine itself does not allow running a VM on a swapping host, so
such hosts will not be used to run new VMs.

Adding folks from the Gluster community to review the swap and other
issues.

Doron