Re: [ovirt-users] swap + crash

Hello, we have big problem with metadata IO error and swap issue with: OS = RHEL - 7 - 1.1503.el7.centos.2.8 kernel = 3.10.0 - 229.4.2.el7.x86_64 KVM Version = 2.1.2 - 23.el7_1.3.1 LIBVIRT = libvirt-1.2.8-16.el7_1.3 gluster = glusterfs-3.7.1-1.el7 ( replica 2 on XFS ) Phys.MEM = 128667 MB total, 23160 MB used, 105507 MB free swap = 398334 MB total, 32049 MB used, 366285 MB free Can anybody get any idea what's wrong ?? System swapping continualy in spite of a lot of free memory available. ( we added large temporary swap space ) Permanent write to /var/log/messages : Journal: End of file while reading data: Input/output error ... and Jun 23 15:46:04 1kvm1 journal: metadata not found: Requested metadata element is not present Jun 23 15:46:06 1kvm1 journal: metadata not found: Requested metadata element is not present Jun 23 15:46:09 1kvm1 kernel: swapper/16: page allocation failure: order:2, mode:0x104020 Jun 23 15:46:09 1kvm1 kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted: G I -------------- 3.10.0-229.4.2.el7.x86_64 #1 Jun 23 15:46:09 1kvm1 kernel: Hardware name: Supermicro X10SRi-F/X10SRi-F, BIOS 1.0a 08/27/2014 Jun 23 15:46:09 1kvm1 kernel: 0000000000104020 04badf98503e3f59 ffff881fff403a00 ffffffff81604eaa Jun 23 15:46:09 1kvm1 kernel: ffff881fff403a90 ffffffff8115c620 ffff88207fffd4e8 ffff881fff403a50 Jun 23 15:46:09 1kvm1 kernel: ffff88207ffd8000 ffff88207ffd8000 0000000000000002 04badf98503e3f59 Jun 23 15:46:09 1kvm1 kernel: Call Trace: Jun 23 15:46:09 1kvm1 kernel: <IRQ> [<ffffffff81604eaa>] dump_stack+0x19/0x1b Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8115c620>] warn_alloc_failed+0x110/0x180 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81160db8>] __alloc_pages_nodemask+0x9a8/0xb90 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8155ab15>] ? tcp_v4_do_rcv+0x1b5/0x470 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8119f549>] alloc_pages_current+0xa9/0x170 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8115b59e>] __get_free_pages+0xe/0x50 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff811aa32e>] kmalloc_order_trace+0x2e/0xa0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff811acbc9>] __kmalloc+0x219/0x230 Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f02ea>] bnx2x_frag_alloc.isra.64+0x2a/0x40 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f15b4>] bnx2x_alloc_rx_data.isra.71+0x54/0x1c0 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f33bd>] bnx2x_rx_int+0x89d/0x1910 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8101bad9>] ? sched_clock+0x9/0x10 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810acd84>] ? task_cputime+0x44/0x80 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810400c2>] ? x86_acpi_suspend_lowlevel+0x12/0x170 Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f485a>] bnx2x_poll+0xfa/0x3c0 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814fcfa2>] net_rx_action+0x152/0x240 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81077bf7>] __do_softirq+0xf7/0x290 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8161671c>] call_softirq+0x1c/0x30 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81015de5>] do_softirq+0x55/0x90 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81077f95>] irq_exit+0x115/0x120 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff816172b8>] do_IRQ+0x58/0xf0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8160c4ad>] common_interrupt+0x6d/0x6d Jun 23 15:46:09 1kvm1 kernel: <EOI> [<ffffffff814aaa32>] ? cpuidle_enter_state+0x52/0xc0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814aaa28>] ? cpuidle_enter_state+0x48/0xc0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814aab65>] cpuidle_idle_call+0xc5/0x200 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8101d21e>] arch_cpu_idle+0xe/0x30 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810c6985>] cpu_startup_entry+0xf5/0x290 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810423ca>] start_secondary+0x1ba/0x230 Jun 23 15:46:16 1kvm1 journal: metadata not found: Requested metadata element is not present Jun 23 15:46:19 1kvm1 journal: metadata not found: Requested metadata element is not present regs. a lot Paf1