Re: [ovirt-users] VMs freezing during heals

I don't think I have the resources to test it meaningfully. I have about 50 vms on my primary storage domain. I might be able to set up a small 50 GB volume and provision 2 or 3 vms running test loads but I'm not sure it would be comparable. I'll give it a try and let you know if I see similar behaviour. On 19 March 2015 at 11:34, Pranith Kumar Karampuri <pkarampu(a)redhat.com&gt; wrote: > Without thinly provisioned lvm. > > Pranith > > On 03/19/2015 08:01 PM, Alastair Neil wrote: > > do you mean raw partitions as bricks or simply with out thin provisioned > lvm? > > > > On 19 March 2015 at 00:32, Pranith Kumar Karampuri <pkarampu(a)redhat.com&gt; > wrote: > >> Could you let me know if you see this problem without lvm as well? >> >> Pranith >> >> On 03/18/2015 08:25 PM, Alastair Neil wrote: >> >> I am in the process of replacing the bricks with thinly provisioned lvs >> yes. >> >> >> >> On 18 March 2015 at 09:35, Pranith Kumar Karampuri <pkarampu(a)redhat.com&gt; >> wrote: >> >>> hi, >>> Are you using thin-lvm based backend on which the bricks are >>> created? >>> >>> Pranith >>> >>> On 03/18/2015 02:05 AM, Alastair Neil wrote: >>> >>> I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There are >>> two virtualisation clusters one with two nehelem nodes and one with four >>> sandybridge nodes. My master storage domain is a GlusterFS backed by a >>> replica 3 gluster volume from 3 of the gluster nodes. The engine is a >>> hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage broviede by >>> nfs from a different gluster volume. All the hosts are CentOS 6.6. >>> >>> vdsm-4.16.10-8.gitc937927.el6 >>>> glusterfs-3.6.2-1.el6 >>>> 2.6.32 - 504.8.1.el6.x86_64 >>> >>> >>> Problems happen when I try to add a new brick or replace a brick >>> eventually the self heal will kill the VMs. In the VM's logs I see kernel >>> hung task messages. >>> >>> Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for >>>> more than 120 seconds. >>>> Mar 12 23:05:16 static1 kernel: Not tainted >>>> 2.6.32-504.3.3.el6.x86_64 #1 >>>> Mar 12 23:05:16 static1 kernel: "echo 0 > >>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>> Mar 12 23:05:16 static1 kernel: nginx D 0000000000000001 0 >>>> 1736 1735 0x00000080 >>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8 0000000000000082 >>>> 0000000000000000 00000000000126c0 >>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500 ffff880037170080 >>>> 0006ce5c85bd9185 ffff88007e5c64d0 >>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0 00000001722b64ba >>>> ffff88007a615098 ffff8800778b1fd8 >>>> Mar 12 23:05:16 static1 kernel: Call Trace: >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>] >>>> schedule_timeout+0x215/0x2e0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a503>] >>>> wait_for_common+0x123/0x180 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81064b90>] ? >>>> default_wake_function+0x0/0x20 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>] ? >>>> _xfs_buf_read+0x46/0x60 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ? >>>> xfs_trans_read_buf+0x197/0x410 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a61d>] >>>> wait_for_completion+0x1d/0x20 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa020ff5b>] >>>> xfs_buf_iowait+0x9b/0x100 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ? >>>> xfs_trans_read_buf+0x197/0x410 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>] >>>> _xfs_buf_read+0x46/0x60 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210b3b>] >>>> xfs_buf_read+0xab/0x100 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] >>>> xfs_trans_read_buf+0x197/0x410 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01ee6a4>] >>>> xfs_imap_to_bp+0x54/0x130 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01f077b>] >>>> xfs_iread+0x7b/0x1b0 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811ab77e>] ? >>>> inode_init_always+0x11e/0x1c0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eb5ee>] >>>> xfs_iget+0x27e/0x6e0 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae1d>] ? >>>> xfs_iunlock+0x5d/0xd0 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0209366>] >>>> xfs_lookup+0xc6/0x110 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0216024>] >>>> xfs_vn_lookup+0x54/0xa0 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119dc65>] >>>> do_lookup+0x1a5/0x230 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119e8f4>] >>>> __link_path_walk+0x7a4/0x1000 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811738e7>] ? >>>> cache_grow+0x217/0x320 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f40a>] >>>> path_walk+0x6a/0xe0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f61b>] >>>> filename_lookup+0x6b/0xc0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811a0747>] >>>> user_path_at+0x57/0xa0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0204e74>] ? >>>> _xfs_trans_commit+0x214/0x2a0 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae3e>] ? >>>> xfs_iunlock+0x7e/0xd0 [xfs] >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193bc0>] >>>> vfs_fstatat+0x50/0xa0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811aaf5d>] ? >>>> touch_atime+0x14d/0x1a0 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d3b>] vfs_stat+0x1b/0x20 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d64>] >>>> sys_newstat+0x24/0x50 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5c87>] ? >>>> audit_syscall_entry+0x1d7/0x200 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5a7e>] ? >>>> __audit_syscall_exit+0x25e/0x290 >>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8100b072>] >>>> system_call_fastpath+0x16/0x1b >>> >>> >>> >>> I am wondering if my volume settings are causing this. Can anyone >>> with more knowledge take a look and let me know: >>> >>> network.remote-dio: on >>>> performance.stat-prefetch: off >>>> performance.io-cache: off >>>> performance.read-ahead: off >>>> performance.quick-read: off >>>> nfs.export-volumes: on >>>> network.ping-timeout: 20 >>>> cluster.self-heal-readdir-size: 64KB >>>> cluster.quorum-type: auto >>>> cluster.data-self-heal-algorithm: diff >>>> cluster.self-heal-window-size: 8 >>>> cluster.heal-timeout: 500 >>>> cluster.self-heal-daemon: on >>>> cluster.entry-self-heal: on >>>> cluster.data-self-heal: on >>>> cluster.metadata-self-heal: on >>>> cluster.readdir-optimize: on >>>> cluster.background-self-heal-count: 20 >>>> cluster.rebalance-stats: on >>>> cluster.min-free-disk: 5% >>>> cluster.eager-lock: enable >>>> storage.owner-uid: 36 >>>> storage.owner-gid: 36 >>>> auth.allow:* >>>> user.cifs: disable >>>> cluster.server-quorum-ratio: 51% >>> >>> >>> Many Thanks, Alastair >>> >>> >>> >>> _______________________________________________ >>> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> > >