This is a multi-part message in MIME format. --------------050800090401020804040607 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello, we have big problem with metadata IO error and swap issue with: OS = RHEL - 7 - 1.1503.el7.centos.2.8 kernel = 3.10.0 - 229.4.2.el7.x86_64 KVM Version = 2.1.2 - 23.el7_1.3.1 LIBVIRT = libvirt-1.2.8-16.el7_1.3 gluster = glusterfs-3.7.1-1.el7 ( replica 2 on XFS ) Phys.MEM = 128667 MB total, 23160 MB used, 105507 MB free swap = 398334 MB total, 32049 MB used, 366285 MB free Can anybody get any idea what's wrong ?? System swapping continualy in spite of a lot of free memory available. ( we added large temporary swap space ) Permanent write to /var/log/messages : Journal: End of file while reading data: Input/output error ... and Jun 23 15:46:04 1kvm1 journal: metadata not found: Requested metadata element is not present Jun 23 15:46:06 1kvm1 journal: metadata not found: Requested metadata element is not present Jun 23 15:46:09 1kvm1 kernel: swapper/16: page allocation failure: order:2, mode:0x104020 Jun 23 15:46:09 1kvm1 kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted: G I -------------- 3.10.0-229.4.2.el7.x86_64 #1 Jun 23 15:46:09 1kvm1 kernel: Hardware name: Supermicro X10SRi-F/X10SRi-F, BIOS 1.0a 08/27/2014 Jun 23 15:46:09 1kvm1 kernel: 0000000000104020 04badf98503e3f59 ffff881fff403a00 ffffffff81604eaa Jun 23 15:46:09 1kvm1 kernel: ffff881fff403a90 ffffffff8115c620 ffff88207fffd4e8 ffff881fff403a50 Jun 23 15:46:09 1kvm1 kernel: ffff88207ffd8000 ffff88207ffd8000 0000000000000002 04badf98503e3f59 Jun 23 15:46:09 1kvm1 kernel: Call Trace: Jun 23 15:46:09 1kvm1 kernel: <IRQ> [<ffffffff81604eaa>] dump_stack+0x19/0x1b Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8115c620>] warn_alloc_failed+0x110/0x180 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81160db8>] __alloc_pages_nodemask+0x9a8/0xb90 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8155ab15>] ? tcp_v4_do_rcv+0x1b5/0x470 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8119f549>] alloc_pages_current+0xa9/0x170 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8115b59e>] __get_free_pages+0xe/0x50 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff811aa32e>] kmalloc_order_trace+0x2e/0xa0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff811acbc9>] __kmalloc+0x219/0x230 Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f02ea>] bnx2x_frag_alloc.isra.64+0x2a/0x40 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f15b4>] bnx2x_alloc_rx_data.isra.71+0x54/0x1c0 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f33bd>] bnx2x_rx_int+0x89d/0x1910 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8101bad9>] ? sched_clock+0x9/0x10 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810acd84>] ? task_cputime+0x44/0x80 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810400c2>] ? x86_acpi_suspend_lowlevel+0x12/0x170 Jun 23 15:46:09 1kvm1 kernel: [<ffffffffa02f485a>] bnx2x_poll+0xfa/0x3c0 [bnx2x] Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814fcfa2>] net_rx_action+0x152/0x240 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81077bf7>] __do_softirq+0xf7/0x290 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8161671c>] call_softirq+0x1c/0x30 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81015de5>] do_softirq+0x55/0x90 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff81077f95>] irq_exit+0x115/0x120 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff816172b8>] do_IRQ+0x58/0xf0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8160c4ad>] common_interrupt+0x6d/0x6d Jun 23 15:46:09 1kvm1 kernel: <EOI> [<ffffffff814aaa32>] ? cpuidle_enter_state+0x52/0xc0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814aaa28>] ? cpuidle_enter_state+0x48/0xc0 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff814aab65>] cpuidle_idle_call+0xc5/0x200 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff8101d21e>] arch_cpu_idle+0xe/0x30 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810c6985>] cpu_startup_entry+0xf5/0x290 Jun 23 15:46:09 1kvm1 kernel: [<ffffffff810423ca>] start_secondary+0x1ba/0x230 Jun 23 15:46:16 1kvm1 journal: metadata not found: Requested metadata element is not present Jun 23 15:46:19 1kvm1 journal: metadata not found: Requested metadata element is not present regs. a lot Paf1 --------------050800090401020804040607 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000066" bgcolor="#FFFFFF"> Hello,<br> we have big problem with metadata IO error and swap issue <br> with: <br> OS =                    RHEL - 7 - 1.1503.el7.centos.2.8<br> kernel =              3.10.0 - 229.4.2.el7.x86_64<br> KVM Version =  2.1.2 - 23.el7_1.3.1<br> LIBVIRT =           libvirt-1.2.8-16.el7_1.3<br> gluster =              glusterfs-3.7.1-1.el7       ( replica 2 on XFS )<br> Phys.MEM =       128667 MB total, 23160 MB used, 105507 MB free<br> swap =                 398334 MB total, 32049 MB used, 366285 MB free<br> <br> Can anybody  get any idea what's wrong ??<br> System swapping continualy in spite of a lot of free memory available. ( we added large temporary swap space  )<br> <br> <br> Permanent write to /var/log/messages :<br> <br> Journal: End of file while reading data: Input/output error<br> ... and <br> Jun 23 15:46:04 1kvm1 journal: metadata not found: Requested metadata element is not present<br> Jun 23 15:46:06 1kvm1 journal: metadata not found: Requested metadata element is not present<br> Jun 23 15:46:09 1kvm1 kernel: swapper/16: page allocation failure: order:2, mode:0x104020<br> Jun 23 15:46:09 1kvm1 kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted: G          I --------------   3.10.0-229.4.2.el7.x86_64 #1<br> Jun 23 15:46:09 1kvm1 kernel: Hardware name: Supermicro X10SRi-F/X10SRi-F, BIOS 1.0a 08/27/2014<br> Jun 23 15:46:09 1kvm1 kernel: 0000000000104020 04badf98503e3f59 ffff881fff403a00 ffffffff81604eaa<br> Jun 23 15:46:09 1kvm1 kernel: ffff881fff403a90 ffffffff8115c620 ffff88207fffd4e8 ffff881fff403a50<br> Jun 23 15:46:09 1kvm1 kernel: ffff88207ffd8000 ffff88207ffd8000 0000000000000002 04badf98503e3f59<br> Jun 23 15:46:09 1kvm1 kernel: Call Trace:<br> Jun 23 15:46:09 1kvm1 kernel: &lt;IRQ&gt;  [&lt;ffffffff81604eaa&gt;] dump_stack+0x19/0x1b<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8115c620&gt;] warn_alloc_failed+0x110/0x180<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff81160db8&gt;] __alloc_pages_nodemask+0x9a8/0xb90<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8155ab15&gt;] ? tcp_v4_do_rcv+0x1b5/0x470<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8119f549&gt;] alloc_pages_current+0xa9/0x170<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8115b59e&gt;] __get_free_pages+0xe/0x50<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff811aa32e&gt;] kmalloc_order_trace+0x2e/0xa0<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff811acbc9&gt;] __kmalloc+0x219/0x230<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffffa02f02ea&gt;] bnx2x_frag_alloc.isra.64+0x2a/0x40 [bnx2x]<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffffa02f15b4&gt;] bnx2x_alloc_rx_data.isra.71+0x54/0x1c0 [bnx2x]<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffffa02f33bd&gt;] bnx2x_rx_int+0x89d/0x1910 [bnx2x]<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8101bad9&gt;] ? sched_clock+0x9/0x10<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff810acd84&gt;] ? task_cputime+0x44/0x80<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff810400c2&gt;] ? x86_acpi_suspend_lowlevel+0x12/0x170<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffffa02f485a&gt;] bnx2x_poll+0xfa/0x3c0 [bnx2x]<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff814fcfa2&gt;] net_rx_action+0x152/0x240<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff81077bf7&gt;] __do_softirq+0xf7/0x290<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8161671c&gt;] call_softirq+0x1c/0x30<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff81015de5&gt;] do_softirq+0x55/0x90<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff81077f95&gt;] irq_exit+0x115/0x120<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff816172b8&gt;] do_IRQ+0x58/0xf0<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8160c4ad&gt;] common_interrupt+0x6d/0x6d<br> Jun 23 15:46:09 1kvm1 kernel: &lt;EOI&gt;  [&lt;ffffffff814aaa32&gt;] ? cpuidle_enter_state+0x52/0xc0<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff814aaa28&gt;] ? cpuidle_enter_state+0x48/0xc0<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff814aab65&gt;] cpuidle_idle_call+0xc5/0x200<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff8101d21e&gt;] arch_cpu_idle+0xe/0x30<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff810c6985&gt;] cpu_startup_entry+0xf5/0x290<br> Jun 23 15:46:09 1kvm1 kernel: [&lt;ffffffff810423ca&gt;] start_secondary+0x1ba/0x230<br> Jun 23 15:46:16 1kvm1 journal: metadata not found: Requested metadata element is not present<br> Jun 23 15:46:19 1kvm1 journal: metadata not found: Requested metadata element is not present<br> <br> regs. a lot<br> Paf1<br> </body> </html> --------------050800090401020804040607--