[ovirt-users] VMs freezing during heals

Fri Mar 20 10:57:26 EDT 2015

Pranith

I have run a pretty straightforward test.  I created a two brick 50 G
replica volume with normal lvm bricks, and installed two servers, one
centos 6.6 and one centos 7.0.  I kicked off bonnie++ on both to generate
some file system activity and then made the volume replica 3.  I saw no
issues on the servers.

Not clear if this is a sufficiently rigorous test and the Volume I have had
issues on is a 3TB volume  with about 2TB used.

-Alastair

On 19 March 2015 at 12:30, Alastair Neil <ajneil.tech at gmail.com> wrote:

> I don't think I have the resources to test it meaningfully.  I have about
> 50 vms on my primary storage domain.  I might be able to set up a small 50
> GB volume and provision 2 or 3 vms running test loads but I'm not sure it
> would be comparable.  I'll give it a try and let you know if I see similar
> behaviour.
>
> On 19 March 2015 at 11:34, Pranith Kumar Karampuri <pkarampu at redhat.com>
> wrote:
>
>>  Without thinly provisioned lvm.
>>
>> Pranith
>>
>> On 03/19/2015 08:01 PM, Alastair Neil wrote:
>>
>> do you mean raw partitions as bricks or simply with out thin provisioned
>> lvm?
>>
>>
>>
>> On 19 March 2015 at 00:32, Pranith Kumar Karampuri <pkarampu at redhat.com>
>> wrote:
>>
>>>  Could you let me know if you see this problem without lvm as well?
>>>
>>> Pranith
>>>
>>> On 03/18/2015 08:25 PM, Alastair Neil wrote:
>>>
>>> I am in the process of replacing the bricks with thinly provisioned lvs
>>> yes.
>>>
>>>
>>>
>>> On 18 March 2015 at 09:35, Pranith Kumar Karampuri <pkarampu at redhat.com>
>>> wrote:
>>>
>>>>  hi,
>>>>       Are you using thin-lvm based backend on which the bricks are
>>>> created?
>>>>
>>>> Pranith
>>>>
>>>> On 03/18/2015 02:05 AM, Alastair Neil wrote:
>>>>
>>>>  I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There are
>>>> two virtualisation clusters one with two nehelem nodes and one with  four
>>>>  sandybridge nodes. My master storage domain is a GlusterFS backed by a
>>>> replica 3 gluster volume from 3 of the gluster nodes.  The engine is a
>>>> hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage broviede by
>>>> nfs from a different gluster volume.  All the hosts are CentOS 6.6.
>>>>
>>>>   vdsm-4.16.10-8.gitc937927.el6
>>>>> glusterfs-3.6.2-1.el6
>>>>> 2.6.32 - 504.8.1.el6.x86_64
>>>>
>>>>
>>>>  Problems happen when I try to add a new brick or replace a brick
>>>> eventually the self heal will kill the VMs. In the VM's logs I see kernel
>>>> hung task messages.
>>>>
>>>>  Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for
>>>>> more than 120 seconds.
>>>>> Mar 12 23:05:16 static1 kernel:      Not tainted
>>>>> 2.6.32-504.3.3.el6.x86_64 #1
>>>>> Mar 12 23:05:16 static1 kernel: "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> Mar 12 23:05:16 static1 kernel: nginx         D 0000000000000001     0
>>>>>  1736   1735 0x00000080
>>>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8 0000000000000082
>>>>> 0000000000000000 00000000000126c0
>>>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500 ffff880037170080
>>>>> 0006ce5c85bd9185 ffff88007e5c64d0
>>>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0 00000001722b64ba
>>>>> ffff88007a615098 ffff8800778b1fd8
>>>>> Mar 12 23:05:16 static1 kernel: Call Trace:
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>]
>>>>> schedule_timeout+0x215/0x2e0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a503>]
>>>>> wait_for_common+0x123/0x180
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81064b90>] ?
>>>>> default_wake_function+0x0/0x20
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>] ?
>>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a61d>]
>>>>> wait_for_completion+0x1d/0x20
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa020ff5b>]
>>>>> xfs_buf_iowait+0x9b/0x100 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>]
>>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210b3b>]
>>>>> xfs_buf_read+0xab/0x100 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>]
>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01ee6a4>]
>>>>> xfs_imap_to_bp+0x54/0x130 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01f077b>]
>>>>> xfs_iread+0x7b/0x1b0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811ab77e>] ?
>>>>> inode_init_always+0x11e/0x1c0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eb5ee>]
>>>>> xfs_iget+0x27e/0x6e0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae1d>] ?
>>>>> xfs_iunlock+0x5d/0xd0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0209366>]
>>>>> xfs_lookup+0xc6/0x110 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0216024>]
>>>>> xfs_vn_lookup+0x54/0xa0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119dc65>]
>>>>> do_lookup+0x1a5/0x230
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119e8f4>]
>>>>> __link_path_walk+0x7a4/0x1000
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811738e7>] ?
>>>>> cache_grow+0x217/0x320
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f40a>]
>>>>> path_walk+0x6a/0xe0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f61b>]
>>>>> filename_lookup+0x6b/0xc0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811a0747>]
>>>>> user_path_at+0x57/0xa0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0204e74>] ?
>>>>> _xfs_trans_commit+0x214/0x2a0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae3e>] ?
>>>>> xfs_iunlock+0x7e/0xd0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193bc0>]
>>>>> vfs_fstatat+0x50/0xa0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811aaf5d>] ?
>>>>> touch_atime+0x14d/0x1a0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d3b>] vfs_stat+0x1b/0x20
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d64>]
>>>>> sys_newstat+0x24/0x50
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5c87>] ?
>>>>> audit_syscall_entry+0x1d7/0x200
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5a7e>] ?
>>>>> __audit_syscall_exit+0x25e/0x290
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8100b072>]
>>>>> system_call_fastpath+0x16/0x1b
>>>>
>>>>
>>>>
>>>>  I am wondering if my volume settings are causing this.  Can anyone
>>>> with more knowledge take a look and let me know:
>>>>
>>>>  network.remote-dio: on
>>>>> performance.stat-prefetch: off
>>>>> performance.io-cache: off
>>>>> performance.read-ahead: off
>>>>> performance.quick-read: off
>>>>> nfs.export-volumes: on
>>>>> network.ping-timeout: 20
>>>>> cluster.self-heal-readdir-size: 64KB
>>>>> cluster.quorum-type: auto
>>>>> cluster.data-self-heal-algorithm: diff
>>>>> cluster.self-heal-window-size: 8
>>>>> cluster.heal-timeout: 500
>>>>> cluster.self-heal-daemon: on
>>>>> cluster.entry-self-heal: on
>>>>> cluster.data-self-heal: on
>>>>> cluster.metadata-self-heal: on
>>>>> cluster.readdir-optimize: on
>>>>> cluster.background-self-heal-count: 20
>>>>> cluster.rebalance-stats: on
>>>>> cluster.min-free-disk: 5%
>>>>> cluster.eager-lock: enable
>>>>> storage.owner-uid: 36
>>>>> storage.owner-gid: 36
>>>>> auth.allow:*
>>>>> user.cifs: disable
>>>>> cluster.server-quorum-ratio: 51%
>>>>
>>>>
>>>>  Many Thanks,  Alastair
>>>>
>>>>
>>>>
>>>>  _______________________________________________
>>>> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150320/4617438b/attachment-0001.html>