5:57 p.m.
Pranith
I have run a pretty straightforward test. I created a two brick 50 G
replica volume with normal lvm bricks, and installed two servers, one
centos 6.6 and one centos 7.0. I kicked off bonnie++ on both to generate
some file system activity and then made the volume replica 3. I saw no
issues on the servers.
Not clear if this is a sufficiently rigorous test and the Volume I have had
issues on is a 3TB volume with about 2TB used.
-Alastair
On 19 March 2015 at 12:30, Alastair Neil <ajneil.tech(a)gmail.com> wrote:
I don't think I have the resources to test it meaningfully. I
have about
50 vms on my primary storage domain. I might be able to set up a small 50
GB volume and provision 2 or 3 vms running test loads but I'm not sure it
would be comparable. I'll give it a try and let you know if I see similar
behaviour.
On 19 March 2015 at 11:34, Pranith Kumar Karampuri <pkarampu(a)redhat.com>
wrote:
> Without thinly provisioned lvm.
>
> Pranith
>
> On 03/19/2015 08:01 PM, Alastair Neil wrote:
>
> do you mean raw partitions as bricks or simply with out thin provisioned
> lvm?
>
>
>
> On 19 March 2015 at 00:32, Pranith Kumar Karampuri <pkarampu(a)redhat.com>
> wrote:
>
>> Could you let me know if you see this problem without lvm as well?
>>
>> Pranith
>>
>> On 03/18/2015 08:25 PM, Alastair Neil wrote:
>>
>> I am in the process of replacing the bricks with thinly provisioned lvs
>> yes.
>>
>>
>>
>> On 18 March 2015 at 09:35, Pranith Kumar Karampuri <pkarampu(a)redhat.com>
>> wrote:
>>
>>> hi,
>>> Are you using thin-lvm based backend on which the bricks are
>>> created?
>>>
>>> Pranith
>>>
>>> On 03/18/2015 02:05 AM, Alastair Neil wrote:
>>>
>>> I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There are
>>> two virtualisation clusters one with two nehelem nodes and one with four
>>> sandybridge nodes. My master storage domain is a GlusterFS backed by a
>>> replica 3 gluster volume from 3 of the gluster nodes. The engine is a
>>> hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage broviede by
>>> nfs from a different gluster volume. All the hosts are CentOS 6.6.
>>>
>>> vdsm-4.16.10-8.gitc937927.el6
>>>> glusterfs-3.6.2-1.el6
>>>> 2.6.32 - 504.8.1.el6.x86_64
>>>
>>>
>>> Problems happen when I try to add a new brick or replace a brick
>>> eventually the self heal will kill the VMs. In the VM's logs I see
kernel
>>> hung task messages.
>>>
>>> Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for
>>>> more than 120 seconds.
>>>> Mar 12 23:05:16 static1 kernel: Not tainted
>>>> 2.6.32-504.3.3.el6.x86_64 #1
>>>> Mar 12 23:05:16 static1 kernel: "echo 0 >
>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>> Mar 12 23:05:16 static1 kernel: nginx D 0000000000000001 0
>>>> 1736 1735 0x00000080
>>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8 0000000000000082
>>>> 0000000000000000 00000000000126c0
>>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500 ffff880037170080
>>>> 0006ce5c85bd9185 ffff88007e5c64d0
>>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0 00000001722b64ba
>>>> ffff88007a615098 ffff8800778b1fd8
>>>> Mar 12 23:05:16 static1 kernel: Call Trace:
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>]
>>>> schedule_timeout+0x215/0x2e0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a503>]
>>>> wait_for_common+0x123/0x180
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81064b90>] ?
>>>> default_wake_function+0x0/0x20
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>] ?
>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a61d>]
>>>> wait_for_completion+0x1d/0x20
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa020ff5b>]
>>>> xfs_buf_iowait+0x9b/0x100 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>]
>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210b3b>]
>>>> xfs_buf_read+0xab/0x100 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>]
>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01ee6a4>]
>>>> xfs_imap_to_bp+0x54/0x130 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01f077b>]
>>>> xfs_iread+0x7b/0x1b0 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811ab77e>] ?
>>>> inode_init_always+0x11e/0x1c0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eb5ee>]
>>>> xfs_iget+0x27e/0x6e0 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae1d>] ?
>>>> xfs_iunlock+0x5d/0xd0 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0209366>]
>>>> xfs_lookup+0xc6/0x110 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0216024>]
>>>> xfs_vn_lookup+0x54/0xa0 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119dc65>]
>>>> do_lookup+0x1a5/0x230
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119e8f4>]
>>>> __link_path_walk+0x7a4/0x1000
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811738e7>] ?
>>>> cache_grow+0x217/0x320
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f40a>]
>>>> path_walk+0x6a/0xe0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f61b>]
>>>> filename_lookup+0x6b/0xc0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811a0747>]
>>>> user_path_at+0x57/0xa0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0204e74>] ?
>>>> _xfs_trans_commit+0x214/0x2a0 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae3e>] ?
>>>> xfs_iunlock+0x7e/0xd0 [xfs]
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193bc0>]
>>>> vfs_fstatat+0x50/0xa0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811aaf5d>] ?
>>>> touch_atime+0x14d/0x1a0
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d3b>]
vfs_stat+0x1b/0x20
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d64>]
>>>> sys_newstat+0x24/0x50
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5c87>] ?
>>>> audit_syscall_entry+0x1d7/0x200
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5a7e>] ?
>>>> __audit_syscall_exit+0x25e/0x290
>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8100b072>]
>>>> system_call_fastpath+0x16/0x1b
>>>
>>>
>>>
>>> I am wondering if my volume settings are causing this. Can anyone
>>> with more knowledge take a look and let me know:
>>>
>>> network.remote-dio: on
>>>> performance.stat-prefetch: off
>>>> performance.io-cache: off
>>>> performance.read-ahead: off
>>>> performance.quick-read: off
>>>> nfs.export-volumes: on
>>>> network.ping-timeout: 20
>>>> cluster.self-heal-readdir-size: 64KB
>>>> cluster.quorum-type: auto
>>>> cluster.data-self-heal-algorithm: diff
>>>> cluster.self-heal-window-size: 8
>>>> cluster.heal-timeout: 500
>>>> cluster.self-heal-daemon: on
>>>> cluster.entry-self-heal: on
>>>> cluster.data-self-heal: on
>>>> cluster.metadata-self-heal: on
>>>> cluster.readdir-optimize: on
>>>> cluster.background-self-heal-count: 20
>>>> cluster.rebalance-stats: on
>>>> cluster.min-free-disk: 5%
>>>> cluster.eager-lock: enable
>>>> storage.owner-uid: 36
>>>> storage.owner-gid: 36
>>>> auth.allow:*
>>>> user.cifs: disable
>>>> cluster.server-quorum-ratio: 51%
>>>
>>>
>>> Many Thanks, Alastair
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing
listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>
>>
>
>
9:57 p.m.
New subject: VMs freezing during heals
--Apple-Mail=_0AF8638A-F586-43E9-B803-1068443CC555
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
What version of gluster are you running on these?
I=E2=80=99ve seen high load during heals bounce my hosted engine around =
due to overall system load, but never pause anything else. Cent 7 combo =
storage/host systems, gluster 3.5.2.
On Mar 20, 2015, at 9:57 AM, Alastair Neil
<ajneil.tech(a)gmail.com> =
wrote:
=20
Pranith
=20
I have run a pretty straightforward test. I created a two brick 50 G =
replica
volume with normal lvm bricks, and installed two servers, one =
centos 6.6 and one centos 7.0. I kicked off bonnie++ on both to =
generate some file system activity and then made the volume replica 3. =
I saw no issues on the servers. =20
=20
Not clear if this is a sufficiently rigorous test and the Volume I =
have had
issues on is a 3TB volume with about 2TB used.
=20
-Alastair
=20
=20
On 19 March 2015 at 12:30, Alastair Neil <ajneil.tech(a)gmail.com =
<mailto:ajneil.tech@gmail.com>> wrote:
I don't think I have the resources to test it meaningfully. I
have =
about 50 vms on my primary storage domain. I might be able to set up a =
small 50 GB volume and provision 2 or 3 vms running test loads but I'm =
not sure it would be comparable. I'll give it a try and let you know if =
I see similar behaviour.
=20
On 19 March 2015 at 11:34, Pranith Kumar Karampuri =
<pkarampu(a)redhat.com
<mailto:pkarampu@redhat.com>> wrote:
Without thinly provisioned lvm.
=20
Pranith
=20
On 03/19/2015 08:01 PM, Alastair Neil wrote:
> do you mean raw partitions as bricks or simply with out thin =
provisioned
lvm?
>=20
>=20
>=20
> On 19 March 2015 at 00:32, Pranith Kumar Karampuri =
<pkarampu(a)redhat.com
<mailto:pkarampu@redhat.com>> wrote:
> Could you let me know if you see this problem without lvm as
well?
>=20
> Pranith
>=20
> On 03/18/2015 08:25 PM, Alastair Neil wrote:
>> I am in the process of replacing the bricks with thinly provisioned =
lvs
yes.
>>=20
>>=20
>>=20
>> On 18 March 2015 at 09:35, Pranith Kumar Karampuri =
<pkarampu(a)redhat.com <mailto:pkarampu@redhat.com>> wrote:
>> hi,
>> Are you using thin-lvm based backend on which the bricks are =
created?
>>=20
>> Pranith
>>=20
>> On 03/18/2015 02:05 AM, Alastair Neil wrote:
>>> I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There =
are
two virtualisation clusters one with two nehelem nodes and one with =
four sandybridge nodes. My master storage domain is a GlusterFS backed =
by a replica 3 gluster volume from 3 of the gluster nodes. The engine =
is a hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage =
broviede by nfs from a different gluster volume. All the hosts are =
CentOS 6.6.
>>>=20
>>> vdsm-4.16.10-8.gitc937927.el6
>>> glusterfs-3.6.2-1.el6
>>> 2.6.32 - 504.8.1.el6.x86_64
>>>=20
>>> Problems happen when I try to add a new brick or replace a brick =
eventually the self heal will kill the VMs. In the VM's logs I see =
kernel hung task messages.=20
>>>=20
>>> Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for =
more than 120 seconds.
>>> Mar 12 23:05:16 static1 kernel: Not tainted =
2.6.32-504.3.3.el6.x86_64 #1
>>> Mar 12 23:05:16 static1 kernel: "echo 0 > =
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Mar 12 23:05:16 static1 kernel: nginx D
0000000000000001 =
0 1736 1735 0x00000080
>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8
0000000000000082 =
0000000000000000 00000000000126c0
>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500
ffff880037170080 =
0006ce5c85bd9185 ffff88007e5c64d0
>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0
00000001722b64ba =
ffff88007a615098 ffff8800778b1fd8
>>> Mar 12 23:05:16 static1 kernel: Call Trace:
>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>] =
schedule_timeout+0x215/0x2e0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8152a503>] =
wait_for_common+0x123/0x180
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81064b90>] ? =
default_wake_function+0x0/0x20
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0210a76>] ? =
_xfs_buf_read+0x46/0x60 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa02063c7>] ? =
xfs_trans_read_buf+0x197/0x410 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8152a61d>] =
wait_for_completion+0x1d/0x20
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa020ff5b>] =
xfs_buf_iowait+0x9b/0x100 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa02063c7>] ? =
xfs_trans_read_buf+0x197/0x410 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0210a76>] =
_xfs_buf_read+0x46/0x60 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0210b3b>] =
xfs_buf_read+0xab/0x100 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa02063c7>] =
xfs_trans_read_buf+0x197/0x410 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01ee6a4>] =
xfs_imap_to_bp+0x54/0x130 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01f077b>] =
xfs_iread+0x7b/0x1b0 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811ab77e>] ? =
inode_init_always+0x11e/0x1c0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01eb5ee>] =
xfs_iget+0x27e/0x6e0 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01eae1d>] ? =
xfs_iunlock+0x5d/0xd0 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0209366>] =
xfs_lookup+0xc6/0x110 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0216024>] =
xfs_vn_lookup+0x54/0xa0 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119dc65>] =
do_lookup+0x1a5/0x230
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119e8f4>] =
__link_path_walk+0x7a4/0x1000
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811738e7>] ? =
cache_grow+0x217/0x320
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119f40a>] =
path_walk+0x6a/0xe0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119f61b>] =
filename_lookup+0x6b/0xc0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811a0747>] =
user_path_at+0x57/0xa0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0204e74>] ? =
_xfs_trans_commit+0x214/0x2a0 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01eae3e>] ? =
xfs_iunlock+0x7e/0xd0 [xfs]
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81193bc0>] =
vfs_fstatat+0x50/0xa0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811aaf5d>] ? =
touch_atime+0x14d/0x1a0
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81193d3b>] =
vfs_stat+0x1b/0x20
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81193d64>] =
sys_newstat+0x24/0x50
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff810e5c87>] ? =
audit_syscall_entry+0x1d7/0x200
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff810e5a7e>] ? =
__audit_syscall_exit+0x25e/0x290
>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8100b072>] =
system_call_fastpath+0x16/0x1b
>>>=20
>>>=20
>>> I am wondering if my volume settings are causing this. Can anyone =
with more knowledge take a look and let me know:
>>>=20
>>> network.remote-dio: on
>>> performance.stat-prefetch: off
>>> performance.io-cache: off
>>> performance.read-ahead: off
>>> performance.quick-read: off
>>> nfs.export-volumes: on
>>> network.ping-timeout: 20
>>> cluster.self-heal-readdir-size: 64KB
>>> cluster.quorum-type: auto
>>> cluster.data-self-heal-algorithm: diff
>>> cluster.self-heal-window-size: 8
>>> cluster.heal-timeout: 500
>>> cluster.self-heal-daemon: on
>>> cluster.entry-self-heal: on
>>> cluster.data-self-heal: on
>>> cluster.metadata-self-heal: on
>>> cluster.readdir-optimize: on
>>> cluster.background-self-heal-count: 20
>>> cluster.rebalance-stats: on
>>> cluster.min-free-disk: 5%
>>> cluster.eager-lock: enable
>>> storage.owner-uid: 36
>>> storage.owner-gid: 36
>>> auth.allow:*
>>> user.cifs: disable
>>> cluster.server-quorum-ratio: 51%
>>>=20
>>> Many Thanks, Alastair
>>>=20
>>>=20
>>>=20
>>> _______________________________________________
>>> Users mailing list
>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>>=20
>>=20
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>>=20
>>=20
>=20
>=20
=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_0AF8638A-F586-43E9-B803-1068443CC555
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">What version of gluster are you running on these?<div =
class=3D""><br class=3D""></div><div
class=3D"">I=E2=80=99ve seen high =
load during heals bounce my hosted engine around due to overall system =
load, but never pause anything else. Cent 7 combo storage/host systems, =
gluster 3.5.2.</div><div class=3D""><br
class=3D""></div><div =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On Mar 20, 2015, at 9:57 AM, Alastair Neil <<a =
href=3D"mailto:ajneil.tech@gmail.com" =
class=3D"">ajneil.tech(a)gmail.com</a>&gt; wrote:</div><br
=
class=3D"Apple-interchange-newline"><div class=3D""><div
dir=3D"ltr" =
class=3D"">Pranith<div class=3D""><br
class=3D""></div><div class=3D"">I =
have run a pretty straightforward test. I created a two brick 50 G =
replica volume with normal lvm bricks, and installed two servers, one =
centos 6.6 and one centos 7.0. I kicked off bonnie++ on both to =
generate some file system activity and then made the volume replica =
3. I saw no issues on the servers. </div><div =
class=3D""><br class=3D""></div><div
class=3D"">Not clear if this is a =
sufficiently rigorous test and the Volume I have had issues on is a 3TB =
volume with about 2TB used.</div><div class=3D""><br =
class=3D""></div><div
class=3D"">-Alastair</div><div class=3D""><br =
class=3D""></div><div class=3D"gmail_extra"><br
class=3D""><div =
class=3D"gmail_quote">On 19 March 2015 at 12:30, Alastair Neil <span =
dir=3D"ltr" class=3D""><<a
href=3D"mailto:ajneil.tech@gmail.com" =
target=3D"_blank"
class=3D"">ajneil.tech(a)gmail.com</a>&gt;</span> =
wrote:<br class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 =
0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div
dir=3D"ltr" =
class=3D"">I don't think I have the resources to test it =
meaningfully. I have about 50 vms on my primary storage =
domain. I might be able to set up a small 50 GB volume and =
provision 2 or 3 vms running test loads but I'm not sure it would be =
comparable. I'll give it a try and let you know if I see similar =
behaviour.</div><div class=3D""><div
class=3D""><div =
class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On 19 =
March 2015 at 11:34, Pranith Kumar Karampuri <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:pkarampu@redhat.com"
target=3D"_blank" =
class=3D"">pkarampu(a)redhat.com</a>&gt;</span> wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex">
=20
=20
=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D"">
Without thinly provisioned lvm.<span class=3D""><font =
color=3D"#888888" class=3D""><br class=3D"">
<br class=3D"">
Pranith</font></span><div class=3D""><div
class=3D""><br class=3D"">
<div class=3D"">On 03/19/2015 08:01 PM, Alastair Neil
wrote:<br class=3D"">
</div>
<blockquote type=3D"cite" class=3D"">
<div dir=3D"ltr" class=3D"">do you mean raw partitions as
bricks =
or simply with
out thin provisioned lvm?
<div class=3D""><br class=3D"">
</div>
<div class=3D""><br class=3D"">
</div>
</div>
<div class=3D"gmail_extra"><br class=3D"">
<div class=3D"gmail_quote">On 19 March 2015 at 00:32, Pranith
Kumar Karampuri <span dir=3D"ltr"
class=3D""><<a =
href=3D"mailto:pkarampu@redhat.com" target=3D"_blank" =
class=3D"">pkarampu(a)redhat.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 =
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D""> Could =
you let me know
if you see this problem without lvm as well?<span =
class=3D""><font color=3D"#888888"
class=3D""><br class=3D"">
<br class=3D"">
Pranith</font></span>
<div class=3D"">
<div class=3D""><br class=3D"">
<div class=3D"">On 03/18/2015 08:25 PM, Alastair Neil =
wrote:<br class=3D"">
</div>
<blockquote type=3D"cite" class=3D"">
<div dir=3D"ltr" class=3D"">I am in the
process of =
replacing the
bricks with thinly provisioned lvs yes.
<div class=3D""><br class=3D"">
</div>
<div class=3D""><br class=3D"">
</div>
</div>
<div class=3D"gmail_extra"><br
class=3D"">
<div class=3D"gmail_quote">On 18 March 2015 at
09:35, Pranith Kumar Karampuri <span dir=3D"ltr" =
class=3D""><<a href=3D"mailto:pkarampu@redhat.com"
target=3D"_blank" =
class=3D"">pkarampu(a)redhat.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
=
class=3D""> hi,<br class=3D"">
Are you
using =
thin-lvm based backend
on which the bricks are created?<br =
class=3D"">
<br class=3D"">
Pranith
<div class=3D"">
<div class=3D""><br
class=3D"">
<div class=3D"">On 03/18/2015 02:05 AM, =
Alastair
Neil wrote:<br class=3D"">
</div>
</div>
</div>
<blockquote type=3D"cite"
class=3D"">
<div class=3D"">
<div class=3D"">
<div dir=3D"ltr" class=3D"">I
have a =
Ovirt cluster
with 6 VM hosts and 4 gluster nodes.
There are two virtualisation
clusters one with two nehelem nodes
and one with four =
sandybridge
nodes. My master storage domain is a
GlusterFS backed by a replica 3
gluster volume from 3 of the gluster
nodes. The engine is a hosted
engine 3.5.1 on 3 of the sandybridge
nodes, with storage broviede by nfs
from a different gluster =
volume.
All the hosts are CentOS 6.6.
<div class=3D""><br
class=3D"">
</div>
<blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px =
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"> vdsm-4.16.10-8.gitc937927.el6<br =
class=3D"">
glusterfs-3.6.2-1.el6<br
class=3D"">=
2.6.32 - =
504.8.1.el6.x86_64</blockquote>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">Problems happen when
=
I try to
add a new brick or replace a brick
eventually the self heal will kill
the VMs. In the VM's logs I see
kernel hung task =
messages. </div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px =
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"><font face=3D"monospace, monospace" =
class=3D"">Mar
12 23:05:16 static1 kernel:
INFO: task nginx:1736 blocked
for more than 120 seconds.<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel: =
Not
tainted
2.6.32-504.3.3.el6.x86_64 =
#1<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel: "echo 0 >
=
/proc/sys/kernel/hung_task_timeout_secs"
disables this message.<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel: nginx =
D 0000000000000001 =
0 1736
1735 0x00000080<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
ffff8800778b17a8
0000000000000082
0000000000000000
00000000000126c0<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
ffff88007e5c6500
ffff880037170080
0006ce5c85bd9185
ffff88007e5c64d0<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
ffff88007a614ae0
00000001722b64ba
ffff88007a615098
ffff8800778b1fd8<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel: Call Trace:<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8152a885>]
=
schedule_timeout+0x215/0x2e0<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8152a503>]
wait_for_common+0x123/0x180<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff81064b90>] ?
=
default_wake_function+0x0/0x20<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa0210a76>] ?
_xfs_buf_read+0x46/0x60 =
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa02063c7>] ?
xfs_trans_read_buf+0x197/0x410
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8152a61d>]
=
wait_for_completion+0x1d/0x20<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa020ff5b>]
xfs_buf_iowait+0x9b/0x100
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa02063c7>] ?
xfs_trans_read_buf+0x197/0x410
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa0210a76>]
_xfs_buf_read+0x46/0x60 =
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa0210b3b>]
xfs_buf_read+0xab/0x100 =
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa02063c7>]
xfs_trans_read_buf+0x197/0x410
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa01ee6a4>]
xfs_imap_to_bp+0x54/0x130
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa01f077b>]
xfs_iread+0x7b/0x1b0 [xfs]<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff811ab77e>] ?
=
inode_init_always+0x11e/0x1c0<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa01eb5ee>]
xfs_iget+0x27e/0x6e0 [xfs]<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa01eae1d>] ?
xfs_iunlock+0x5d/0xd0 [xfs]<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa0209366>]
xfs_lookup+0xc6/0x110 [xfs]<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa0216024>]
xfs_vn_lookup+0x54/0xa0 =
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8119dc65>]
do_lookup+0x1a5/0x230<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8119e8f4>]
=
__link_path_walk+0x7a4/0x1000<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff811738e7>] ?
cache_grow+0x217/0x320<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8119f40a>]
path_walk+0x6a/0xe0<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8119f61b>]
filename_lookup+0x6b/0xc0<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff811a0747>]
user_path_at+0x57/0xa0<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa0204e74>] ?
_xfs_trans_commit+0x214/0x2a0
[xfs]<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffffa01eae3e>] ?
xfs_iunlock+0x7e/0xd0 [xfs]<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff81193bc0>]
vfs_fstatat+0x50/0xa0<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff811aaf5d>] ?
touch_atime+0x14d/0x1a0<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff81193d3b>]
vfs_stat+0x1b/0x20<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff81193d64>]
sys_newstat+0x24/0x50<br =
class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff810e5c87>] ?
=
audit_syscall_entry+0x1d7/0x200<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff810e5a7e>] ?
=
__audit_syscall_exit+0x25e/0x290<br class=3D"">
</font><font face=3D"monospace,
monospace" class=3D"">Mar 12
=
23:05:16
static1 kernel:
[<ffffffff8100b072>]
=
system_call_fastpath+0x16/0x1b</font></blockquote>
</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">I am wondering if my
=
volume
settings are causing this. =
Can
anyone with more knowledge take a
look and let me know:</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0px 0px 0px =
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"><font face=3D"monospace, monospace" =
class=3D"">network.remote-dio:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">performance.stat-prefetch:
off<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">performance.io-cache:
off<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">performance.read-ahead:
off<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">performance.quick-read:
off<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">nfs.export-volumes:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">network.ping-timeout:
20<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.self-heal-readdir-size:
64KB<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.quorum-type:
auto<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.data-self-heal-algorithm:
diff<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.self-heal-window-size:
8<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.heal-timeout:
500<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.self-heal-daemon:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.entry-self-heal:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.data-self-heal:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.metadata-self-heal:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.readdir-optimize:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.background-self-heal-count:
20<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.rebalance-stats:
on<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.min-free-disk:
5%<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.eager-lock:
enable<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">storage.owner-uid:
36<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">storage.owner-gid:
36<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">auth.allow:*<br class=3D"">
</font><font face=3D"monospace,
monospace"
class=3D"">user.cifs:=
disable<br class=3D"">
</font><font face=3D"monospace,
monospace" =
class=3D"">cluster.server-quorum-ratio:
51%</font></blockquote>
</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">Many Thanks, =
Alastair</div>
<div class=3D""><br
class=3D"">
</div>
</div>
<br class=3D"">
<fieldset class=3D""></fieldset>
<br class=3D"">
</div>
</div>
<pre =
class=3D"">_______________________________________________
Users mailing list
<a href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a>
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users" =
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<...
</pre>
</blockquote>
<br class=3D"">
</div>
<br class=3D"">
_______________________________________________<br class=3D"">
Users mailing list<br class=3D"">
<a href=3D"mailto:Users@ovirt.org" =
target=3D"_blank" class=3D"">Users(a)ovirt.org</a><br
class=3D"">
<a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... =
class=3D"">
<br class=3D"">
</blockquote>
</div>
<br class=3D"">
</div>
</blockquote>
<br class=3D"">
</div>
</div>
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</blockquote>
<br class=3D"">
</div></div></div>
</blockquote></div><br class=3D""></div>
</div></div></blockquote></div><br
class=3D""></div></div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_0AF8638A-F586-43E9-B803-1068443CC555--
10:22 p.m.
New subject: VMs freezing during heals
CentOS 6.6
vdsm-4.16.10-8.gitc937927.el6
glusterfs-3.6.2-1.el6
2.6.32 - 504.8.1.el6.x86_64
moved to 3.6 specifically to get the snapshotting feature, hence my desire
to migrate to thinly provisioned lvm bricks.
On 20 March 2015 at 14:57, Darrell Budic <budic(a)onholyground.com> wrote:
What version of gluster are you running on these?
I’ve seen high load during heals bounce my hosted engine around due to
overall system load, but never pause anything else. Cent 7 combo
storage/host systems, gluster 3.5.2.
On Mar 20, 2015, at 9:57 AM, Alastair Neil <ajneil.tech(a)gmail.com> wrote:
Pranith
I have run a pretty straightforward test. I created a two brick 50 G
replica volume with normal lvm bricks, and installed two servers, one
centos 6.6 and one centos 7.0. I kicked off bonnie++ on both to generate
some file system activity and then made the volume replica 3. I saw no
issues on the servers.
Not clear if this is a sufficiently rigorous test and the Volume I have
had issues on is a 3TB volume with about 2TB used.
-Alastair
On 19 March 2015 at 12:30, Alastair Neil <ajneil.tech(a)gmail.com> wrote:
> I don't think I have the resources to test it meaningfully. I have about
> 50 vms on my primary storage domain. I might be able to set up a small 50
> GB volume and provision 2 or 3 vms running test loads but I'm not sure it
> would be comparable. I'll give it a try and let you know if I see similar
> behaviour.
>
> On 19 March 2015 at 11:34, Pranith Kumar Karampuri <pkarampu(a)redhat.com>
> wrote:
>
>> Without thinly provisioned lvm.
>>
>> Pranith
>>
>> On 03/19/2015 08:01 PM, Alastair Neil wrote:
>>
>> do you mean raw partitions as bricks or simply with out thin provisioned
>> lvm?
>>
>>
>>
>> On 19 March 2015 at 00:32, Pranith Kumar Karampuri <pkarampu(a)redhat.com>
>> wrote:
>>
>>> Could you let me know if you see this problem without lvm as well?
>>>
>>> Pranith
>>>
>>> On 03/18/2015 08:25 PM, Alastair Neil wrote:
>>>
>>> I am in the process of replacing the bricks with thinly provisioned lvs
>>> yes.
>>>
>>>
>>>
>>> On 18 March 2015 at 09:35, Pranith Kumar Karampuri <pkarampu(a)redhat.com
>>> > wrote:
>>>
>>>> hi,
>>>> Are you using thin-lvm based backend on which the bricks are
>>>> created?
>>>>
>>>> Pranith
>>>>
>>>> On 03/18/2015 02:05 AM, Alastair Neil wrote:
>>>>
>>>> I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There
>>>> are two virtualisation clusters one with two nehelem nodes and one with
>>>> four sandybridge nodes. My master storage domain is a GlusterFS backed
by
>>>> a replica 3 gluster volume from 3 of the gluster nodes. The engine is a
>>>> hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage broviede
by
>>>> nfs from a different gluster volume. All the hosts are CentOS 6.6.
>>>>
>>>> vdsm-4.16.10-8.gitc937927.el6
>>>>> glusterfs-3.6.2-1.el6
>>>>> 2.6.32 - 504.8.1.el6.x86_64
>>>>
>>>>
>>>> Problems happen when I try to add a new brick or replace a brick
>>>> eventually the self heal will kill the VMs. In the VM's logs I see
kernel
>>>> hung task messages.
>>>>
>>>> Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for
>>>>> more than 120 seconds.
>>>>> Mar 12 23:05:16 static1 kernel: Not tainted
>>>>> 2.6.32-504.3.3.el6.x86_64 #1
>>>>> Mar 12 23:05:16 static1 kernel: "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> Mar 12 23:05:16 static1 kernel: nginx D 0000000000000001
>>>>> 0 1736 1735 0x00000080
>>>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8 0000000000000082
>>>>> 0000000000000000 00000000000126c0
>>>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500 ffff880037170080
>>>>> 0006ce5c85bd9185 ffff88007e5c64d0
>>>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0 00000001722b64ba
>>>>> ffff88007a615098 ffff8800778b1fd8
>>>>> Mar 12 23:05:16 static1 kernel: Call Trace:
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>]
>>>>> schedule_timeout+0x215/0x2e0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a503>]
>>>>> wait_for_common+0x123/0x180
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81064b90>] ?
>>>>> default_wake_function+0x0/0x20
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>] ?
>>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a61d>]
>>>>> wait_for_completion+0x1d/0x20
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa020ff5b>]
>>>>> xfs_buf_iowait+0x9b/0x100 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>]
>>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210b3b>]
>>>>> xfs_buf_read+0xab/0x100 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>]
>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01ee6a4>]
>>>>> xfs_imap_to_bp+0x54/0x130 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01f077b>]
>>>>> xfs_iread+0x7b/0x1b0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811ab77e>] ?
>>>>> inode_init_always+0x11e/0x1c0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eb5ee>]
>>>>> xfs_iget+0x27e/0x6e0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae1d>] ?
>>>>> xfs_iunlock+0x5d/0xd0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0209366>]
>>>>> xfs_lookup+0xc6/0x110 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0216024>]
>>>>> xfs_vn_lookup+0x54/0xa0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119dc65>]
>>>>> do_lookup+0x1a5/0x230
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119e8f4>]
>>>>> __link_path_walk+0x7a4/0x1000
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811738e7>] ?
>>>>> cache_grow+0x217/0x320
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f40a>]
>>>>> path_walk+0x6a/0xe0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f61b>]
>>>>> filename_lookup+0x6b/0xc0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811a0747>]
>>>>> user_path_at+0x57/0xa0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0204e74>] ?
>>>>> _xfs_trans_commit+0x214/0x2a0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae3e>] ?
>>>>> xfs_iunlock+0x7e/0xd0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193bc0>]
>>>>> vfs_fstatat+0x50/0xa0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811aaf5d>] ?
>>>>> touch_atime+0x14d/0x1a0
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d3b>]
>>>>> vfs_stat+0x1b/0x20
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d64>]
>>>>> sys_newstat+0x24/0x50
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5c87>] ?
>>>>> audit_syscall_entry+0x1d7/0x200
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5a7e>] ?
>>>>> __audit_syscall_exit+0x25e/0x290
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8100b072>]
>>>>> system_call_fastpath+0x16/0x1b
>>>>
>>>>
>>>>
>>>> I am wondering if my volume settings are causing this. Can anyone
>>>> with more knowledge take a look and let me know:
>>>>
>>>> network.remote-dio: on
>>>>> performance.stat-prefetch: off
>>>>> performance.io-cache: off
>>>>> performance.read-ahead: off
>>>>> performance.quick-read: off
>>>>> nfs.export-volumes: on
>>>>> network.ping-timeout: 20
>>>>> cluster.self-heal-readdir-size: 64KB
>>>>> cluster.quorum-type: auto
>>>>> cluster.data-self-heal-algorithm: diff
>>>>> cluster.self-heal-window-size: 8
>>>>> cluster.heal-timeout: 500
>>>>> cluster.self-heal-daemon: on
>>>>> cluster.entry-self-heal: on
>>>>> cluster.data-self-heal: on
>>>>> cluster.metadata-self-heal: on
>>>>> cluster.readdir-optimize: on
>>>>> cluster.background-self-heal-count: 20
>>>>> cluster.rebalance-stats: on
>>>>> cluster.min-free-disk: 5%
>>>>> cluster.eager-lock: enable
>>>>> storage.owner-uid: 36
>>>>> storage.owner-gid: 36
>>>>> auth.allow:*
>>>>> user.cifs: disable
>>>>> cluster.server-quorum-ratio: 51%
>>>>
>>>>
>>>> Many Thanks, Alastair
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing
listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>>
>>
>>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
11:04 p.m.
New subject: VMs freezing during heals
Any follow up on this?
Are there known issues using a replica 3 glsuter datastore with lvm thin
provisioned bricks?
On 20 March 2015 at 15:22, Alastair Neil <ajneil.tech(a)gmail.com> wrote:
CentOS 6.6
> vdsm-4.16.10-8.gitc937927.el6
> glusterfs-3.6.2-1.el6
> 2.6.32 - 504.8.1.el6.x86_64
moved to 3.6 specifically to get the snapshotting feature, hence my desire
to migrate to thinly provisioned lvm bricks.
On 20 March 2015 at 14:57, Darrell Budic <budic(a)onholyground.com> wrote:
> What version of gluster are you running on these?
>
> I’ve seen high load during heals bounce my hosted engine around due to
> overall system load, but never pause anything else. Cent 7 combo
> storage/host systems, gluster 3.5.2.
>
>
> On Mar 20, 2015, at 9:57 AM, Alastair Neil <ajneil.tech(a)gmail.com> wrote:
>
> Pranith
>
> I have run a pretty straightforward test. I created a two brick 50 G
> replica volume with normal lvm bricks, and installed two servers, one
> centos 6.6 and one centos 7.0. I kicked off bonnie++ on both to generate
> some file system activity and then made the volume replica 3. I saw no
> issues on the servers.
>
> Not clear if this is a sufficiently rigorous test and the Volume I have
> had issues on is a 3TB volume with about 2TB used.
>
> -Alastair
>
>
> On 19 March 2015 at 12:30, Alastair Neil <ajneil.tech(a)gmail.com> wrote:
>
>> I don't think I have the resources to test it meaningfully. I have
>> about 50 vms on my primary storage domain. I might be able to set up a
>> small 50 GB volume and provision 2 or 3 vms running test loads but I'm not
>> sure it would be comparable. I'll give it a try and let you know if I see
>> similar behaviour.
>>
>> On 19 March 2015 at 11:34, Pranith Kumar Karampuri <pkarampu(a)redhat.com>
>> wrote:
>>
>>> Without thinly provisioned lvm.
>>>
>>> Pranith
>>>
>>> On 03/19/2015 08:01 PM, Alastair Neil wrote:
>>>
>>> do you mean raw partitions as bricks or simply with out thin
>>> provisioned lvm?
>>>
>>>
>>>
>>> On 19 March 2015 at 00:32, Pranith Kumar Karampuri <pkarampu(a)redhat.com
>>> > wrote:
>>>
>>>> Could you let me know if you see this problem without lvm as well?
>>>>
>>>> Pranith
>>>>
>>>> On 03/18/2015 08:25 PM, Alastair Neil wrote:
>>>>
>>>> I am in the process of replacing the bricks with thinly provisioned
>>>> lvs yes.
>>>>
>>>>
>>>>
>>>> On 18 March 2015 at 09:35, Pranith Kumar Karampuri <
>>>> pkarampu(a)redhat.com> wrote:
>>>>
>>>>> hi,
>>>>> Are you using thin-lvm based backend on which the bricks are
>>>>> created?
>>>>>
>>>>> Pranith
>>>>>
>>>>> On 03/18/2015 02:05 AM, Alastair Neil wrote:
>>>>>
>>>>> I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There
>>>>> are two virtualisation clusters one with two nehelem nodes and one
with
>>>>> four sandybridge nodes. My master storage domain is a GlusterFS
backed by
>>>>> a replica 3 gluster volume from 3 of the gluster nodes. The engine
is a
>>>>> hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage
broviede by
>>>>> nfs from a different gluster volume. All the hosts are CentOS 6.6.
>>>>>
>>>>> vdsm-4.16.10-8.gitc937927.el6
>>>>>> glusterfs-3.6.2-1.el6
>>>>>> 2.6.32 - 504.8.1.el6.x86_64
>>>>>
>>>>>
>>>>> Problems happen when I try to add a new brick or replace a brick
>>>>> eventually the self heal will kill the VMs. In the VM's logs I
see kernel
>>>>> hung task messages.
>>>>>
>>>>> Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for
>>>>>> more than 120 seconds.
>>>>>> Mar 12 23:05:16 static1 kernel: Not tainted
>>>>>> 2.6.32-504.3.3.el6.x86_64 #1
>>>>>> Mar 12 23:05:16 static1 kernel: "echo 0 >
>>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
>>>>>> Mar 12 23:05:16 static1 kernel: nginx D 0000000000000001
>>>>>> 0 1736 1735 0x00000080
>>>>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8
0000000000000082
>>>>>> 0000000000000000 00000000000126c0
>>>>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500
ffff880037170080
>>>>>> 0006ce5c85bd9185 ffff88007e5c64d0
>>>>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0
00000001722b64ba
>>>>>> ffff88007a615098 ffff8800778b1fd8
>>>>>> Mar 12 23:05:16 static1 kernel: Call Trace:
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>]
>>>>>> schedule_timeout+0x215/0x2e0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a503>]
>>>>>> wait_for_common+0x123/0x180
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81064b90>] ?
>>>>>> default_wake_function+0x0/0x20
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>] ?
>>>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a61d>]
>>>>>> wait_for_completion+0x1d/0x20
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa020ff5b>]
>>>>>> xfs_buf_iowait+0x9b/0x100 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>] ?
>>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210a76>]
>>>>>> _xfs_buf_read+0x46/0x60 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0210b3b>]
>>>>>> xfs_buf_read+0xab/0x100 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa02063c7>]
>>>>>> xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01ee6a4>]
>>>>>> xfs_imap_to_bp+0x54/0x130 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01f077b>]
>>>>>> xfs_iread+0x7b/0x1b0 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811ab77e>] ?
>>>>>> inode_init_always+0x11e/0x1c0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eb5ee>]
>>>>>> xfs_iget+0x27e/0x6e0 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae1d>] ?
>>>>>> xfs_iunlock+0x5d/0xd0 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0209366>]
>>>>>> xfs_lookup+0xc6/0x110 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0216024>]
>>>>>> xfs_vn_lookup+0x54/0xa0 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119dc65>]
>>>>>> do_lookup+0x1a5/0x230
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119e8f4>]
>>>>>> __link_path_walk+0x7a4/0x1000
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811738e7>] ?
>>>>>> cache_grow+0x217/0x320
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f40a>]
>>>>>> path_walk+0x6a/0xe0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8119f61b>]
>>>>>> filename_lookup+0x6b/0xc0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811a0747>]
>>>>>> user_path_at+0x57/0xa0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa0204e74>] ?
>>>>>> _xfs_trans_commit+0x214/0x2a0 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffffa01eae3e>] ?
>>>>>> xfs_iunlock+0x7e/0xd0 [xfs]
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193bc0>]
>>>>>> vfs_fstatat+0x50/0xa0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff811aaf5d>] ?
>>>>>> touch_atime+0x14d/0x1a0
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d3b>]
>>>>>> vfs_stat+0x1b/0x20
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff81193d64>]
>>>>>> sys_newstat+0x24/0x50
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5c87>] ?
>>>>>> audit_syscall_entry+0x1d7/0x200
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff810e5a7e>] ?
>>>>>> __audit_syscall_exit+0x25e/0x290
>>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8100b072>]
>>>>>> system_call_fastpath+0x16/0x1b
>>>>>
>>>>>
>>>>>
>>>>> I am wondering if my volume settings are causing this. Can anyone
>>>>> with more knowledge take a look and let me know:
>>>>>
>>>>> network.remote-dio: on
>>>>>> performance.stat-prefetch: off
>>>>>> performance.io-cache: off
>>>>>> performance.read-ahead: off
>>>>>> performance.quick-read: off
>>>>>> nfs.export-volumes: on
>>>>>> network.ping-timeout: 20
>>>>>> cluster.self-heal-readdir-size: 64KB
>>>>>> cluster.quorum-type: auto
>>>>>> cluster.data-self-heal-algorithm: diff
>>>>>> cluster.self-heal-window-size: 8
>>>>>> cluster.heal-timeout: 500
>>>>>> cluster.self-heal-daemon: on
>>>>>> cluster.entry-self-heal: on
>>>>>> cluster.data-self-heal: on
>>>>>> cluster.metadata-self-heal: on
>>>>>> cluster.readdir-optimize: on
>>>>>> cluster.background-self-heal-count: 20
>>>>>> cluster.rebalance-stats: on
>>>>>> cluster.min-free-disk: 5%
>>>>>> cluster.eager-lock: enable
>>>>>> storage.owner-uid: 36
>>>>>> storage.owner-gid: 36
>>>>>> auth.allow:*
>>>>>> user.cifs: disable
>>>>>> cluster.server-quorum-ratio: 51%
>>>>>
>>>>>
>>>>> Many Thanks, Alastair
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing
listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
4:57 p.m.
New subject: VMs freezing during heals
This is a multi-part message in MIME format.
--------------030308050700080206070001
Content-Type: text/plain;
charset="windows-1252"
Content-Transfer-Encoding: quoted-printable
On 04/03/2015 10=3A04 PM=2C Alastair Neil wrote=3A
=3E Any follow up on this=3F
=3E
=3E Are there known issues using a replica 3 glsuter datastore with lvm
=3E thin provisioned bricks=3F
=3E
=3E On 20 March 2015 at 15=3A22=2C Alastair Neil =3Cajneil=2Etech=40gmail=
=2Ecom
=3E =3Cmailto=3Aajneil=2Etech=40gmail=2Ecom=3E=3E wrote=3A
=3E
=3E CentOS 6=2E6
=3E =20
=3E
=3E vdsm-4=2E16=2E10-8=2Egitc937927=2Eel6
=3E glusterfs-3=2E6=2E2-1=2Eel6
=3E 2=2E6=2E32 - 504=2E8=2E1=2Eel6=2Ex86=5F64
=3E
=3E
=3E moved to 3=2E6 specifically to get the snapshotting feature=2C henc=
e
=3E my desire to migrate to thinly provisioned lvm bricks=2E
=3E
Well on the glusterfs mailinglist there have been discussions=3A
=3E 3=2E6=2E2 is a major release and introduces some new features in cluste=
r
=3E wide concept=2E Additionally it is not stable yet=2E
=3E
=3E
=3E On 20 March 2015 at 14=3A57=2C Darrell Budic =3Cbudic=40onholygroun=
d=2Ecom
=3E =3Cmailto=3Abudic=40onholyground=2Ecom=3E=3E wrote=3A
=3E
=3E What version of gluster are you running on these=3F
=3E
=3E I=92ve seen high load during heals bounce my hosted engine
=3E around due to overall system load=2C but never pause anything=
=3E else=2E Cent 7 combo storage/host systems=2C gluster 3=2E5=2E2=
=2E
=3E
=3E
=3E=3E On Mar 20=2C 2015=2C at 9=3A57 AM=2C Alastair Neil
=3E=3E =3Cajneil=2Etech=40gmail=2Ecom =3Cmailto=3Aajneil=2Etech=40g=
mail=2Ecom=3E=3E wrote=3A
=3E=3E
=3E=3E Pranith
=3E=3E
=3E=3E I have run a pretty straightforward test=2E I created a two=
=3E=3E brick 50 G replica volume with normal lvm bricks=2C and
=3E=3E installed two servers=2C one centos 6=2E6 and one centos 7=
=2E0=2E I
=3E=3E kicked off bonnie++ on both to generate some file system
=3E=3E activity and then made the volume replica 3=2E I saw no iss=
ues
=3E=3E on the servers=2E =20
=3E=3E
=3E=3E Not clear if this is a sufficiently rigorous test and the
=3E=3E Volume I have had issues on is a 3TB volume with about 2TB=
used=2E
=3E=3E
=3E=3E -Alastair
=3E=3E
=3E=3E
=3E=3E On 19 March 2015 at 12=3A30=2C Alastair Neil
=3E=3E =3Cajneil=2Etech=40gmail=2Ecom =3Cmailto=3Aajneil=2Etech=40g=
mail=2Ecom=3E=3E wrote=3A
=3E=3E
=3E=3E I don=27t think I have the resources to test it
=3E=3E meaningfully=2E I have about 50 vms on my primary stora=
ge
=3E=3E domain=2E I might be able to set up a small 50 GB volum=
e
=3E=3E and provision 2 or 3 vms running test loads but I=27m no=
t
=3E=3E sure it would be comparable=2E I=27ll give it a try and=
let
=3E=3E you know if I see similar behaviour=2E
=3E=3E
=3E=3E On 19 March 2015 at 11=3A34=2C Pranith Kumar Karampuri=
=3E=3E =3Cpkarampu=40redhat=2Ecom =3Cmailto=3Apkarampu=40redhat=
=2Ecom=3E=3E wrote=3A
=3E=3E
=3E=3E Without thinly provisioned lvm=2E
=3E=3E
=3E=3E Pranith
=3E=3E
=3E=3E On 03/19/2015 08=3A01 PM=2C Alastair Neil wrote=3A=
=3E=3E=3E do you mean raw partitions as bricks or simply wi=
th
=3E=3E=3E out thin provisioned lvm=3F
=3E=3E=3E
=3E=3E=3E
=3E=3E=3E
=3E=3E=3E On 19 March 2015 at 00=3A32=2C Pranith Kumar Kara=
mpuri
=3E=3E=3E =3Cpkarampu=40redhat=2Ecom =3Cmailto=3Apkarampu=
=40redhat=2Ecom=3E=3E
=3E=3E=3E wrote=3A
=3E=3E=3E
=3E=3E=3E Could you let me know if you see this problem=
=3E=3E=3E without lvm as well=3F
=3E=3E=3E
=3E=3E=3E Pranith
=3E=3E=3E
=3E=3E=3E On 03/18/2015 08=3A25 PM=2C Alastair Neil wro=
te=3A
=3E=3E=3E=3E I am in the process of replacing the brick=
s
=3E=3E=3E=3E with thinly provisioned lvs yes=2E
=3E=3E=3E=3E
=3E=3E=3E=3E
=3E=3E=3E=3E
=3E=3E=3E=3E On 18 March 2015 at 09=3A35=2C Pranith Kum=
ar
=3E=3E=3E=3E Karampuri =3Cpkarampu=40redhat=2Ecom
=3E=3E=3E=3E =3Cmailto=3Apkarampu=40redhat=2Ecom=3E=3E=
wrote=3A
=3E=3E=3E=3E
=3E=3E=3E=3E hi=2C
=3E=3E=3E=3E Are you using thin-lvm based bac=
kend
=3E=3E=3E=3E on which the bricks are created=3F
=3E=3E=3E=3E
=3E=3E=3E=3E Pranith
=3E=3E=3E=3E
=3E=3E=3E=3E On 03/18/2015 02=3A05 AM=2C Alastair N=
eil wrote=3A
=3E=3E=3E=3E=3E I have a Ovirt cluster with 6 VM ho=
sts and
=3E=3E=3E=3E=3E 4 gluster nodes=2E There are two
=3E=3E=3E=3E=3E virtualisation clusters one with tw=
o
=3E=3E=3E=3E=3E nehelem nodes and one with four
=3E=3E=3E=3E=3E sandybridge nodes=2E My master sto=
rage
=3E=3E=3E=3E=3E domain is a GlusterFS backed by a r=
eplica
=3E=3E=3E=3E=3E 3 gluster volume from 3 of the glus=
ter
=3E=3E=3E=3E=3E nodes=2E The engine is a hosted en=
gine
=3E=3E=3E=3E=3E 3=2E5=2E1 on 3 of the sandybridge n=
odes=2C with
=3E=3E=3E=3E=3E storage broviede by nfs from a diff=
erent
=3E=3E=3E=3E=3E gluster volume=2E All the hosts ar=
e CentOS
=3E=3E=3E=3E=3E 6=2E6=2E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E vdsm-4=2E16=2E10-8=2Egitc93792=
7=2Eel6
=3E=3E=3E=3E=3E glusterfs-3=2E6=2E2-1=2Eel6
=3E=3E=3E=3E=3E 2=2E6=2E32 - 504=2E8=2E1=2Eel6=
=2Ex86=5F64
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E Problems happen when I try to add a=
new
=3E=3E=3E=3E=3E brick or replace a brick eventually=
the
=3E=3E=3E=3E=3E self heal will kill the VMs=2E In t=
he VM=27s
=3E=3E=3E=3E=3E logs I see kernel hung task message=
s=2E=20
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A INFO=3A
=3E=3E=3E=3E=3E task nginx=3A1736 blocked for m=
ore than
=3E=3E=3E=3E=3E 120 seconds=2E
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A =20
=3E=3E=3E=3E=3E Not tainted 2=2E6=2E32-504=2E3=
=2E3=2Eel6=2Ex86=5F64 =231
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A =22echo
=3E=3E=3E=3E=3E 0 =3E
=3E=3E=3E=3E=3E /proc/sys/kernel/hung=5Ftask=5F=
timeout=5Fsecs=22
=3E=3E=3E=3E=3E disables this message=2E
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A nginx
=3E=3E=3E=3E=3E D 0000000000000001 =20=
0 1736
=3E=3E=3E=3E=3E 1735 0x00000080
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E ffff8800778b17a8 00000000000000=
82
=3E=3E=3E=3E=3E 0000000000000000 00000000000126=
c0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E ffff88007e5c6500 ffff8800371700=
80
=3E=3E=3E=3E=3E 0006ce5c85bd9185 ffff88007e5c64=
d0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E ffff88007a614ae0 00000001722b64=
ba
=3E=3E=3E=3E=3E ffff88007a615098 ffff8800778b1f=
d8
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A Call
=3E=3E=3E=3E=3E Trace=3A
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8152a885=3E=5D
=3E=3E=3E=3E=3E schedule=5Ftimeout+0x215/0x2e0=
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8152a503=3E=5D
=3E=3E=3E=3E=3E wait=5Ffor=5Fcommon+0x123/0x180=
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff81064b90=3E=5D=20=
=3F
=3E=3E=3E=3E=3E default=5Fwake=5Ffunction+0x0/0=
x20
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa0210a76=3E=5D=20=
=3F
=3E=3E=3E=3E=3E =5Fxfs=5Fbuf=5Fread+0x46/0x60=
=5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa02063c7=3E=5D=20=
=3F
=3E=3E=3E=3E=3E xfs=5Ftrans=5Fread=5Fbuf+0x197/=
0x410 =5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8152a61d=3E=5D
=3E=3E=3E=3E=3E wait=5Ffor=5Fcompletion+0x1d/0x=
20
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa020ff5b=3E=5D
=3E=3E=3E=3E=3E xfs=5Fbuf=5Fiowait+0x9b/0x100=
=5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa02063c7=3E=5D=20=
=3F
=3E=3E=3E=3E=3E xfs=5Ftrans=5Fread=5Fbuf+0x197/=
0x410 =5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa0210a76=3E=5D
=3E=3E=3E=3E=3E =5Fxfs=5Fbuf=5Fread+0x46/0x60=
=5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa0210b3b=3E=5D
=3E=3E=3E=3E=3E xfs=5Fbuf=5Fread+0xab/0x100=20=
=5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa02063c7=3E=5D
=3E=3E=3E=3E=3E xfs=5Ftrans=5Fread=5Fbuf+0x197/=
0x410 =5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa01ee6a4=3E=5D
=3E=3E=3E=3E=3E xfs=5Fimap=5Fto=5Fbp+0x54/0x130=
=5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa01f077b=3E=5D
=3E=3E=3E=3E=3E xfs=5Firead+0x7b/0x1b0 =5Bxfs=
=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff811ab77e=3E=5D=20=
=3F
=3E=3E=3E=3E=3E inode=5Finit=5Falways+0x11e/0x1=
c0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa01eb5ee=3E=5D
=3E=3E=3E=3E=3E xfs=5Figet+0x27e/0x6e0 =5Bxfs=
=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa01eae1d=3E=5D=20=
=3F
=3E=3E=3E=3E=3E xfs=5Fiunlock+0x5d/0xd0 =5Bxfs=
=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa0209366=3E=5D
=3E=3E=3E=3E=3E xfs=5Flookup+0xc6/0x110 =5Bxfs=
=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa0216024=3E=5D
=3E=3E=3E=3E=3E xfs=5Fvn=5Flookup+0x54/0xa0=20=
=5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8119dc65=3E=5D do=
=5Flookup+0x1a5/0x230
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8119e8f4=3E=5D
=3E=3E=3E=3E=3E =5F=5Flink=5Fpath=5Fwalk+0x7a4/=
0x1000
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff811738e7=3E=5D=20=
=3F
=3E=3E=3E=3E=3E cache=5Fgrow+0x217/0x320
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8119f40a=3E=5D pa=
th=5Fwalk+0x6a/0xe0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8119f61b=3E=5D
=3E=3E=3E=3E=3E filename=5Flookup+0x6b/0xc0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff811a0747=3E=5D
=3E=3E=3E=3E=3E user=5Fpath=5Fat+0x57/0xa0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa0204e74=3E=5D=20=
=3F
=3E=3E=3E=3E=3E =5Fxfs=5Ftrans=5Fcommit+0x214/0=
x2a0 =5Bxfs=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffffa01eae3e=3E=5D=20=
=3F
=3E=3E=3E=3E=3E xfs=5Fiunlock+0x7e/0xd0 =5Bxfs=
=5D
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff81193bc0=3E=5D vf=
s=5Ffstatat+0x50/0xa0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff811aaf5d=3E=5D=20=
=3F
=3E=3E=3E=3E=3E touch=5Fatime+0x14d/0x1a0
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff81193d3b=3E=5D vf=
s=5Fstat+0x1b/0x20
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff81193d64=3E=5D sy=
s=5Fnewstat+0x24/0x50
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff810e5c87=3E=5D=20=
=3F
=3E=3E=3E=3E=3E audit=5Fsyscall=5Fentry+0x1d7/0=
x200
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff810e5a7e=3E=5D=20=
=3F
=3E=3E=3E=3E=3E =5F=5Faudit=5Fsyscall=5Fexit+0x=
25e/0x290
=3E=3E=3E=3E=3E Mar 12 23=3A05=3A16 static1 ker=
nel=3A
=3E=3E=3E=3E=3E =5B=3Cffffffff8100b072=3E=5D
=3E=3E=3E=3E=3E system=5Fcall=5Ffastpath+0x16/0=
x1b
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E I am wondering if my volume setting=
s are
=3E=3E=3E=3E=3E causing this=2E Can anyone with mo=
re
=3E=3E=3E=3E=3E knowledge take a look and let me kn=
ow=3A
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E network=2Eremote-dio=3A on
=3E=3E=3E=3E=3E performance=2Estat-prefetch=3A=
off
=3E=3E=3E=3E=3E performance=2Eio-cache=3A off=
=3E=3E=3E=3E=3E performance=2Eread-ahead=3A off=
=3E=3E=3E=3E=3E performance=2Equick-read=3A off=
=3E=3E=3E=3E=3E nfs=2Eexport-volumes=3A on
=3E=3E=3E=3E=3E network=2Eping-timeout=3A 20
=3E=3E=3E=3E=3E cluster=2Eself-heal-readdir-siz=
e=3A 64KB
=3E=3E=3E=3E=3E cluster=2Equorum-type=3A auto=
=3E=3E=3E=3E=3E cluster=2Edata-self-heal-algori=
thm=3A diff
=3E=3E=3E=3E=3E cluster=2Eself-heal-window-size=
=3A 8
=3E=3E=3E=3E=3E cluster=2Eheal-timeout=3A 500=
=3E=3E=3E=3E=3E cluster=2Eself-heal-daemon=3A o=
n
=3E=3E=3E=3E=3E cluster=2Eentry-self-heal=3A on=
=3E=3E=3E=3E=3E cluster=2Edata-self-heal=3A on=
=3E=3E=3E=3E=3E cluster=2Emetadata-self-heal=3A=
on
=3E=3E=3E=3E=3E cluster=2Ereaddir-optimize=3A o=
n
=3E=3E=3E=3E=3E cluster=2Ebackground-self-heal-=
count=3A 20
=3E=3E=3E=3E=3E cluster=2Erebalance-stats=3A on=
=3E=3E=3E=3E=3E cluster=2Emin-free-disk=3A 5=25=
=3E=3E=3E=3E=3E cluster=2Eeager-lock=3A enable=
=3E=3E=3E=3E=3E storage=2Eowner-uid=3A 36
=3E=3E=3E=3E=3E storage=2Eowner-gid=3A 36
=3E=3E=3E=3E=3E auth=2Eallow=3A*
=3E=3E=3E=3E=3E user=2Ecifs=3A disable
=3E=3E=3E=3E=3E cluster=2Eserver-quorum-ratio=
=3A 51=25
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E Many Thanks=2C Alastair
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E
=3E=3E=3E=3E=3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
=3E=3E=3E=3E=3E Users mailing list
=3E=3E=3E=3E=3E Users=40ovirt=2Eorg =3Cmailto=3AUse=
rs=40ovirt=2Eorg=3E
=3E=3E=3E=3E=3E http=3A//lists=2Eovirt=2Eorg/mailma=
n/listinfo/users
=3E=3E=3E=3E
=3E=3E=3E=3E
=3E=3E=3E=3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
=3E=3E=3E=3E Users mailing list
=3E=3E=3E=3E Users=40ovirt=2Eorg =3Cmailto=3AUsers=
=40ovirt=2Eorg=3E
=3E=3E=3E=3E http=3A//lists=2Eovirt=2Eorg/mailman/l=
istinfo/users
=3E=3E=3E=3E
=3E=3E=3E=3E
=3E=3E=3E
=3E=3E=3E
=3E=3E
=3E=3E
=3E=3E
=3E=3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F
=3E=3E Users mailing list
=3E=3E Users=40ovirt=2Eorg =3Cmailto=3AUsers=40ovirt=2Eorg=3E
=3E=3E http=3A//lists=2Eovirt=2Eorg/mailman/listinfo/users
=3E
=3E
=3E
=3E
=3E
=3E
=3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
=3E Users mailing list
=3E Users=40ovirt=2Eorg
=3E http=3A//lists=2Eovirt=2Eorg/mailman/listinfo/users
Met vriendelijke groet=2C With kind regards=2C
Jorick Astrego
Netbulae Virtualization Experts=20
----------------
=09Tel=3A 053 20 30 270 =09info=40netbulae=2Eeu =09Staalsteden 4-3A =09KvK=
08198180
=09Fax=3A 053 20 30 271 =09www=2Enetbulae=2Eeu =097547 TA Enschede =09BTW=
NL821234584B01
----------------
--------------030308050700080206070001
Content-Type: text/html;
charset="windows-1252"
Content-Transfer-Encoding: quoted-printable
=3Chtml=3E
=3Chead=3E
=3Cmeta content=3D=22text/html=3B charset=3Dwindows-1252=22
http-equiv=3D=22Content-Type=22=3E
=3C/head=3E
=3Cbody bgcolor=3D=22=23FFFFFF=22 text=3D=22=23000000=22=3E
=3Cbr=3E
=3Cbr=3E
=3Cdiv class=3D=22moz-cite-prefix=22=3EOn 04/03/2015 10=3A04 PM=2C Alas=
tair Neil
wrote=3A=3Cbr=3E
=3C/div=3E
=3Cblockquote
cite=3D=22mid=3ACA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w=40mail=
=2Egmail=2Ecom=22
type=3D=22cite=22=3E
=3Cdiv dir=3D=22ltr=22=3EAny follow up on this=3F
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv=3E=A0Are there known issues using a replica 3 glsuter datast=
ore
with lvm thin provisioned bricks=3F=3C/div=3E
=3C/div=3E
=3Cdiv class=3D=22gmail=5Fextra=22=3E=3Cbr=3E
=3Cdiv class=3D=22gmail=5Fquote=22=3EOn 20 March 2015 at 15=3A22=2C=
Alastair
Neil =3Cspan dir=3D=22ltr=22=3E=26lt=3B=3Ca moz-do-not-send=3D=22=
true=22
href=3D=22mailto=3Aajneil=2Etech=40gmail=2Ecom=22 target=3D=
=22=5Fblank=22=3Eajneil=2Etech=40gmail=2Ecom=3C/a=3E=26gt=3B=3C/span=3E
wrote=3A=3Cbr=3E
=3Cblockquote class=3D=22gmail=5Fquote=22 style=3D=22margin=3A0 0=
0
=2E8ex=3Bborder-left=3A1px =23ccc solid=3Bpadding-left=3A1ex=22=
=3E
=3Cdiv dir=3D=22ltr=22=3E
=3Cdiv=3ECentOS 6=2E6=3C/div=3E
=3Cspan class=3D=22=22=3E
=3Cdiv=3E=A0=3C/div=3E
=3Cblockquote class=3D=22gmail=5Fquote=22
style=3D=22font-size=3A13px=3Bmargin=3A0px 0px 0px
0=2E8ex=3Bborder-left-width=3A1px=3Bborder-left-color=3Argb=28204=2C204=2C2=
04=29=3Bborder-left-style=3Asolid=3Bpadding-left=3A1ex=22=3E=A0vdsm-4=2E16=
=2E10-8=2Egitc937927=2Eel6=3Cbr=3E
glusterfs-3=2E6=2E2-1=2Eel6=3Cbr=3E
2=2E6=2E32 - 504=2E8=2E1=2Eel6=2Ex86=5F64=3C/blockquote=
=3E
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3C/span=3E
=3Cdiv=3Emoved to 3=2E6 specifically to get the snapshotting=
feature=2C hence my desire to migrate to thinly
provisioned lvm bricks=2E=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3Cbr=3E
=3Cbr=3E
Well on the glusterfs mailinglist there have been discussions=3A=3Cbr=
=3E
=3Cbr=3E
=3Cbr=3E
=3Cblockquote type=3D=22cite=22=3E3=2E6=2E2 is a major release and intr=
oduces some
new features in cluster wide concept=2E Additionally it is not
stable yet=2E=3C/blockquote=3E
=3Cbr=3E
=3Cbr=3E
=3Cbr=3E
=3Cbr=3E
=3Cblockquote
cite=3D=22mid=3ACA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w=40mail=
=2Egmail=2Ecom=22
type=3D=22cite=22=3E
=3Cdiv class=3D=22gmail=5Fextra=22=3E
=3Cdiv class=3D=22gmail=5Fquote=22=3E
=3Cblockquote class=3D=22gmail=5Fquote=22 style=3D=22margin=3A0 0=
0
=2E8ex=3Bborder-left=3A1px =23ccc solid=3Bpadding-left=3A1ex=22=
=3E
=3Cdiv dir=3D=22ltr=22=3E
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv class=3D=22gmail=5Fextra=22=3E=3Cbr=3E
=3Cdiv class=3D=22gmail=5Fquote=22=3EOn 20 March 2015 at 14=
=3A57=2C
Darrell Budic =3Cspan dir=3D=22ltr=22=3E=26lt=3B=3Ca
moz-do-not-send=3D=22true=22
href=3D=22mailto=3Abudic=40onholyground=2Ecom=22
target=3D=22=5Fblank=22=3Ebudic=40onholyground=2Ecom=
=3C/a=3E=26gt=3B=3C/span=3E
wrote=3A=3Cbr=3E
=3Cblockquote class=3D=22gmail=5Fquote=22 style=3D=22marg=
in=3A0 0 0
=2E8ex=3Bborder-left=3A1px =23ccc solid=3Bpadding-left=
=3A1ex=22=3E
=3Cdiv style=3D=22word-wrap=3Abreak-word=22=3EWhat vers=
ion of
gluster are you running on these=3F
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv=3EI=92ve seen high load during heals bounce my=
hosted engine around due to overall system load=2C=
but never pause anything else=2E Cent 7 combo
storage/host systems=2C gluster 3=2E5=2E2=2E=3C/div=
=3E
=3Cdiv=3E
=3Cdiv=3E
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv=3E=3Cbr=3E
=3Cdiv=3E
=3Cblockquote type=3D=22cite=22=3E
=3Cdiv=3EOn Mar 20=2C 2015=2C at 9=3A57 AM=
=2C
Alastair Neil =26lt=3B=3Ca
moz-do-not-send=3D=22true=22
href=3D=22mailto=3Aajneil=2Etech=40gmai=
l=2Ecom=22
target=3D=22=5Fblank=22=3Eajneil=2Etech=
=40gmail=2Ecom=3C/a=3E=26gt=3B
wrote=3A=3C/div=3E
=3Cbr=3E
=3Cdiv=3E
=3Cdiv dir=3D=22ltr=22=3EPranith
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv=3EI have run a pretty
straightforward test=2E=A0 I created=
a
two brick 50 G replica volume with
normal lvm bricks=2C and installed
two servers=2C one centos 6=2E6 and=
one centos 7=2E0=2E=A0 I kicked off=
bonnie++ on both to generate some
file system activity and then made
the volume replica 3=2E=A0 I saw no=
issues on the servers=2E =A0=A0=3C/di=
v=3E
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv=3ENot clear if this is a
sufficiently rigorous test and the
Volume I have had issues on is a
3TB volume =A0with about 2TB used=2E=
=3C/div=3E
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv=3E-Alastair=3C/div=3E
=3Cdiv=3E=3Cbr=3E
=3C/div=3E
=3Cdiv class=3D=22gmail=5Fextra=22=3E=
=3Cbr=3E
=3Cdiv class=3D=22gmail=5Fquote=22=3E=
On 19
March 2015 at 12=3A30=2C Alastair=
Neil =3Cspan dir=3D=22ltr=22=3E=26l=
t=3B=3Ca
moz-do-not-send=3D=22true=22
href=3D=22mailto=3Aajneil=2Etec=
h=40gmail=2Ecom=22
target=3D=22=5Fblank=22=3Eajnei=
l=2Etech=40gmail=2Ecom=3C/a=3E=26gt=3B=3C/span=3E
wrote=3A=3Cbr=3E
=3Cblockquote class=3D=22gmail=5Fqu=
ote=22
style=3D=22margin=3A0 0 0
=2E8ex=3Bborder-left=3A1px =23ccc=
solid=3Bpadding-left=3A1ex=22=3E=
=3Cdiv dir=3D=22ltr=22=3EI don=27=
t think I
have the resources to test
it meaningfully=2E=A0 I have
about 50 vms on my primary
storage domain=2E=A0 I might be=
able to set up a small 50 GB
volume and provision 2 or 3
vms running test loads but
I=27m not sure it would be
comparable=2E=A0 I=27ll give it=
a
try and let you know if I
see similar behaviour=2E=3C/div=
=3E
=3Cdiv=3E
=3Cdiv=3E
=3Cdiv class=3D=22gmail=5Fext=
ra=22=3E=3Cbr=3E
=3Cdiv class=3D=22gmail=5Fq=
uote=22=3EOn
19 March 2015 at
11=3A34=2C Pranith Kumar=
Karampuri =3Cspan
dir=3D=22ltr=22=3E=26lt=
=3B=3Ca
moz-do-not-send=3D=22=
true=22
href=3D=22mailto=3Apkarampu=40redhat=2Ecom=22 target=3D=22=5Fblank=22=3Epka=
rampu=40redhat=2Ecom=3C/a=3E=26gt=3B=3C/span=3E
wrote=3A=3Cbr=3E
=3Cblockquote
class=3D=22gmail=5Fquot=
e=22
style=3D=22margin=3A0 0=
0
=2E8ex=3Bborder-left=3A=
1px
=23ccc
solid=3Bpadding-left=3A=
1ex=22=3E
=3Cdiv text=3D=22=23000=
000=22
bgcolor=3D=22=23FFFFF=
F=22=3E
Without thinly
provisioned lvm=2E=3C=
span=3E=3Cfont
color=3D=22=23888888=22=3E=3Cbr=3E
=3Cbr=3E
Pranith=3C/font=
=3E=3C/span=3E
=3Cdiv=3E
=3Cdiv=3E=3Cbr=3E=
=3Cdiv=3EOn
03/19/2015
08=3A01 PM=2C
Alastair Neil
wrote=3A=3Cbr=3E=
=3C/div=3E
=3Cblockquote
type=3D=22cite=22=
=3E
=3Cdiv dir=3D=22l=
tr=22=3Edo
you mean raw
partitions as
bricks or
simply with
out thin
provisioned
lvm=3F
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3C/div=3E
=3Cdiv
class=3D=22gmail=
=5Fextra=22=3E=3Cbr=3E
=3Cdiv
class=3D=22gmail=
=5Fquote=22=3EOn
19 March 2015
at 00=3A32=2C
Pranith Kumar
Karampuri =3Cspan=
dir=3D=22ltr=22=
=3E=26lt=3B=3Ca
moz-do-not-send=3D=22true=22 href=3D=22mailto=3Apkarampu=40redhat=2Ecom=22=
target=3D=22=5Fblank=22=3Epkarampu=40redhat=2Ecom=3C/a=3E=26gt=3B=3C/span=
=3E
wrote=3A=3Cbr=3E=
=3Cblockquote
class=3D=22gmail=
=5Fquote=22
style=3D=22margin=
=3A0
0 0
=2E8ex=3Bborder-l=
eft=3A1px
=23ccc
solid=3Bpadding-l=
eft=3A1ex=22=3E
=3Cdiv
text=3D=22=230000=
00=22
bgcolor=3D=22=23FFFFFF=22=3E Could you let me know if you see this problem=
without
lvm as well=3F=3C=
span=3E=3Cfont
color=3D=22=23888888=22=3E=3Cbr=3E
=3Cbr=3E
Pranith=3C/font=
=3E=3C/span=3E
=3Cdiv=3E
=3Cdiv=3E=3Cbr=3E=
=3Cdiv=3EOn
03/18/2015
08=3A25 PM=2C
Alastair Neil
wrote=3A=3Cbr=3E=
=3C/div=3E
=3Cblockquote
type=3D=22cite=22=
=3E
=3Cdiv dir=3D=22l=
tr=22=3EI
am in the
process of
replacing the
bricks with
thinly
provisioned
lvs yes=2E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3C/div=3E
=3Cdiv
class=3D=22gmail=
=5Fextra=22=3E=3Cbr=3E
=3Cdiv
class=3D=22gmail=
=5Fquote=22=3EOn
18 March 2015
at 09=3A35=2C
Pranith Kumar
Karampuri =3Cspan=
dir=3D=22ltr=22=
=3E=26lt=3B=3Ca
moz-do-not-send=3D=22true=22 href=3D=22mailto=3Apkarampu=40redhat=2Ecom=22=
target=3D=22=5Fblank=22=3Epkarampu=40redhat=2Ecom=3C/a=3E=26gt=3B=3C/span=
=3E
wrote=3A=3Cbr=3E=
=3Cblockquote
class=3D=22gmail=
=5Fquote=22
style=3D=22margin=
=3A0
0 0
=2E8ex=3Bborder-l=
eft=3A1px
=23ccc
solid=3Bpadding-l=
eft=3A1ex=22=3E
=3Cdiv
text=3D=22=230000=
00=22
bgcolor=3D=22=23FFFFFF=22=3E hi=2C=3Cbr=3E
=A0=A0=A0=A0=A0 A=
re you
using thin-lvm
based backend
on which the
bricks are
created=3F=3Cbr=
=3E
=3Cbr=3E
Pranith
=3Cdiv=3E
=3Cdiv=3E=3Cbr=3E=
=3Cdiv=3EOn
03/18/2015
02=3A05 AM=2C
Alastair Neil
wrote=3A=3Cbr=3E=
=3C/div=3E
=3C/div=3E
=3C/div=3E
=3Cblockquote
type=3D=22cite=22=
=3E
=3Cdiv=3E
=3Cdiv=3E
=3Cdiv dir=3D=22l=
tr=22=3EI
have a Ovirt
cluster with 6
VM hosts and 4
gluster nodes=2E=
There are two
virtualisation
clusters one
with two
nehelem nodes
and one with
=A0four
=A0sandybridge
nodes=2E My
master storage
domain is a
GlusterFS
backed by a
replica 3
gluster volume
from 3 of the
gluster
nodes=2E=A0 The=
engine is a
hosted engine
3=2E5=2E1 on 3 of=
the
sandybridge
nodes=2C with
storage
broviede by
nfs from a
different
gluster
volume=2E=A0 All=
the hosts are
CentOS 6=2E6=2E=
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cblockquote
class=3D=22gmail=
=5Fquote=22
style=3D=22margin=
=3A0px
0px 0px
0=2E8ex=3Bborder-left-width=3A1px=3Bborder-left-color=3Argb=28204=2C204=2C2=
04=29=3Bborder-left-style=3Asolid=3Bpadding-left=3A1ex=22=3E=A0vdsm-4=2E16=
=2E10-8=2Egitc937927=2Eel6=3Cbr=3E
glusterfs-3=2E6=2E2-1=2Eel6=3Cbr=3E
2=2E6=2E32 -
504=2E8=2E1=2Eel6=
=2Ex86=5F64=3C/blockquote=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3EProblems=
happen when I
try to add a
new brick or
replace a
brick
eventually the
self heal will
kill the VMs=2E=
In the VM=27s
logs I see
kernel hung
task
messages=2E=A0=3C=
/div=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3E
=3Cblockquote
class=3D=22gmail=
=5Fquote=22
style=3D=22margin=
=3A0px
0px 0px
0=2E8ex=3Bborder-left-width=3A1px=3Bborder-left-color=3Argb=28204=2C204=2C2=
04=29=3Bborder-left-style=3Asolid=3Bpadding-left=3A1ex=22=3E=3Cfont
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A INFO=3A=
task
nginx=3A1736
blocked for
more than 120
seconds=2E=3Cbr=
=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A =A0=20=
=A0
=A0Not tainted
2=2E6=2E32-504=2E=
3=2E3=2Eel6=2Ex86=5F64
=231=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A =22echo=
0 =26gt=3B
/proc/sys/kernel/=
hung=5Ftask=5Ftimeout=5Fsecs=22
disables this
message=2E=3Cbr=
=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A nginx=
=A0 =A0 =A0 =A0 D=
0000000000000001=
=A0 =A0 0 =A01736=
=A0
1735
0x00000080=3Cbr=
=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
ffff8800778b17a8=
0000000000000082=
0000000000000000=
00000000000126c0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
ffff88007e5c6500=
ffff880037170080=
0006ce5c85bd9185=
ffff88007e5c64d0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
ffff88007a614ae0=
00000001722b64ba=
ffff88007a615098=
ffff8800778b1fd8=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A Call
Trace=3A=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8152a885=26gt=3B=5D
schedule=5Ftimeout+0x215/0x2e0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8152a503=26gt=3B=5D
wait=5Ffor=5Fcommon+0x123/0x180=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff81064b90=26gt=3B=5D
=3F
default=5Fwake=5F=
function+0x0/0x20=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa0210a76=26gt=3B=5D
=3F
=5Fxfs=5Fbuf=5Fre=
ad+0x46/0x60
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa02063c7=26gt=3B=5D
=3F
xfs=5Ftrans=5Frea=
d=5Fbuf+0x197/0x410
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8152a61d=26gt=3B=5D
wait=5Ffor=5Fcompletion+0x1d/0x20=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa020ff5b=26gt=3B=5D
xfs=5Fbuf=5Fiowai=
t+0x9b/0x100
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa02063c7=26gt=3B=5D
=3F
xfs=5Ftrans=5Frea=
d=5Fbuf+0x197/0x410
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa0210a76=26gt=3B=5D
=5Fxfs=5Fbuf=5Fre=
ad+0x46/0x60
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa0210b3b=26gt=3B=5D
xfs=5Fbuf=5Fread+=
0xab/0x100
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa02063c7=26gt=3B=5D
xfs=5Ftrans=5Frea=
d=5Fbuf+0x197/0x410
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa01ee6a4=26gt=3B=5D
xfs=5Fimap=5Fto=
=5Fbp+0x54/0x130
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa01f077b=26gt=3B=5D
xfs=5Firead+0x7b/=
0x1b0
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff811ab77e=26gt=3B=5D
=3F
inode=5Finit=5Fal=
ways+0x11e/0x1c0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa01eb5ee=26gt=3B=5D
xfs=5Figet+0x27e/=
0x6e0
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa01eae1d=26gt=3B=5D
=3F
xfs=5Fiunlock+0x5=
d/0xd0
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa0209366=26gt=3B=5D
xfs=5Flookup+0xc6=
/0x110
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa0216024=26gt=3B=5D
xfs=5Fvn=5Flookup=
+0x54/0xa0
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8119dc65=26gt=3B=5D
do=5Flookup+0x1a5/0x230=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8119e8f4=26gt=3B=5D
=5F=5Flink=5Fpath=5Fwalk+0x7a4/0x1000=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff811738e7=26gt=3B=5D
=3F
cache=5Fgrow+0x21=
7/0x320=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8119f40a=26gt=3B=5D
path=5Fwalk+0x6a/0xe0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8119f61b=26gt=3B=5D
filename=5Flookup+0x6b/0xc0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff811a0747=26gt=3B=5D
user=5Fpath=5Fat+0x57/0xa0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa0204e74=26gt=3B=5D
=3F
=5Fxfs=5Ftrans=5F=
commit+0x214/0x2a0
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ffa01eae3e=26gt=3B=5D
=3F
xfs=5Fiunlock+0x7=
e/0xd0
=5Bxfs=5D=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff81193bc0=26gt=3B=5D
vfs=5Ffstatat+0x50/0xa0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff811aaf5d=26gt=3B=5D
=3F
touch=5Fatime+0x1=
4d/0x1a0=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff81193d3b=26gt=3B=5D
vfs=5Fstat+0x1b/0x20=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff81193d64=26gt=3B=5D
sys=5Fnewstat+0x24/0x50=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff810e5c87=26gt=3B=5D
=3F
audit=5Fsyscall=
=5Fentry+0x1d7/0x200=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff810e5a7e=26gt=3B=5D
=3F
=5F=5Faudit=5Fsys=
call=5Fexit+0x25e/0x290=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3EMa=
r
12 23=3A05=3A16=
static1
kernel=3A
=5B=26lt=3Bffffff=
ff8100b072=26gt=3B=5D
system=5Fcall=5Ffastpath+0x16/0x1b=3C/font=3E=3C/blockquote=3E
=3C/div=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3EI am
wondering if
my volume
settings are
causing this=2E=
=A0
Can anyone
with more
knowledge take
a look and let
me know=3A=3C/div=
=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3E
=3Cblockquote
class=3D=22gmail=
=5Fquote=22
style=3D=22margin=
=3A0px
0px 0px
0=2E8ex=3Bborder-left-width=3A1px=3Bborder-left-color=3Argb=28204=2C204=2C2=
04=29=3Bborder-left-style=3Asolid=3Bpadding-left=3A1ex=22=3E=3Cfont
face=3D=22monospa=
ce=2C
monospace=22=3Ene=
twork=2Eremote-dio=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Epe=
rformance=2Estat-prefetch=3A
off=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Epe=
rformance=2Eio-cache=3A
off=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Epe=
rformance=2Eread-ahead=3A
off=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Epe=
rformance=2Equick-read=3A
off=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Enf=
s=2Eexport-volumes=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ene=
twork=2Eping-timeout=3A
20=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eself-heal-readdir-size=3A
64KB=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Equorum-type=3A
auto=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Edata-self-heal-algorithm=3A
diff=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eself-heal-window-size=3A
8=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eheal-timeout=3A
500=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eself-heal-daemon=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eentry-self-heal=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Edata-self-heal=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Emetadata-self-heal=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Ereaddir-optimize=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Ebackground-self-heal-count=3A
20=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Erebalance-stats=3A
on=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Emin-free-disk=3A
5=25=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eeager-lock=3A
enable=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Est=
orage=2Eowner-uid=3A
36=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Est=
orage=2Eowner-gid=3A
36=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Eau=
th=2Eallow=3A*=3Cbr=3E
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Eus=
er=2Ecifs=3A
disable=3Cbr=3E=
=3C/font=3E=3Cfon=
t
face=3D=22monospa=
ce=2C
monospace=22=3Ecl=
uster=2Eserver-quorum-ratio=3A
51=25=3C/font=3E=
=3C/blockquote=3E
=3C/div=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3Cdiv=3EMany
Thanks=2C
=A0Alastair=3C/di=
v=3E
=3Cdiv=3E=3Cbr=3E=
=3C/div=3E
=3C/div=3E
=3Cbr=3E
=3Cfieldset=3E=3C=
/fieldset=3E
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3Cpre=3E=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Users mailing list
=3Ca moz-do-not-send=3D=22true=22 href=3D=22mailto=3AUsers=40ovirt=2Eorg=22=
target=3D=22=5Fblank=22=3EUsers=40ovirt=2Eorg=3C/a=3E
=3Ca moz-do-not-send=3D=22true=22 href=3D=22http=3A//lists=2Eovirt=2Eorg/ma=
ilman/listinfo/users=22 target=3D=22=5Fblank=22=3Ehttp=3A//lists=2Eovirt=2E=
org/mailman/listinfo/users=3C/a=3E
=3C/pre=3E
=3C/blockquote=3E=
=3Cbr=3E
=3C/div=3E
=3Cbr=3E
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=3Cbr=3E=
Users mailing
list=3Cbr=3E
=3Ca
moz-do-not-send=
=3D=22true=22
href=3D=22mailto=3AUsers=40ovirt=2Eorg=22 target=3D=22=5Fblank=22=3EUsers=
=40ovirt=2Eorg=3C/a=3E=3Cbr=3E
=3Ca
moz-do-not-send=
=3D=22true=22
href=3D=22http=3A//lists=2Eovirt=2Eorg/mailman/listinfo/users=22 target=3D=
=22=5Fblank=22=3Ehttp=3A//lists=2Eovirt=2Eorg/mailman/listinfo/users=3C/a=
=3E=3Cbr=3E
=3Cbr=3E
=3C/blockquote=3E=
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3C/blockquote=3E=
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E=
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3C/blockquote=3E=
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=3Cbr=3E=
Users mailing list=3Cbr=3E
=3Ca moz-do-not-send=3D=22true=22
href=3D=22mailto=3AUsers=40ovirt=2Eorg=
=22
target=3D=22=5Fblank=22=3EUsers=40ovirt=
=2Eorg=3C/a=3E=3Cbr=3E
=3Ca moz-do-not-send=3D=22true=22
href=3D=22http=3A//lists=2Eovirt=2Eorg/=
mailman/listinfo/users=22
target=3D=22=5Fblank=22=3Ehttp=3A//list=
s=2Eovirt=2Eorg/mailman/listinfo/users=3C/a=3E=3Cbr=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3Cdiv class=3D=22HOEnZb=22=3E
=3Cdiv class=3D=22h5=22=3E=3Cbr=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3Cbr=3E
=3Cbr=3E
=3Cblockquote
cite=3D=22mid=3ACA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w=40mail=
=2Egmail=2Ecom=22
type=3D=22cite=22=3E
=3Cdiv class=3D=22gmail=5Fextra=22=3E
=3Cdiv class=3D=22gmail=5Fquote=22=3E
=3Cblockquote class=3D=22gmail=5Fquote=22 style=3D=22margin=3A0 0=
0
=2E8ex=3Bborder-left=3A1px =23ccc solid=3Bpadding-left=3A1ex=22=
=3E
=3Cdiv class=3D=22HOEnZb=22=3E
=3Cdiv class=3D=22h5=22=3E
=3C/div=3E
=3C/div=3E
=3C/blockquote=3E
=3C/div=3E
=3Cbr=3E
=3C/div=3E
=3Cbr=3E
=3Cfieldset class=3D=22mimeAttachmentHeader=22=3E=3C/fieldset=3E
=3Cbr=3E
=3Cpre wrap=3D=22=22=3E=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
=5F=5F=5F=5F=5F=5F=5F
Users mailing list
=3Ca class=3D=22moz-txt-link-abbreviated=22 href=3D=22mailto=3AUsers=40ovir=
t=2Eorg=22=3EUsers=40ovirt=2Eorg=3C/a=3E
=3Ca class=3D=22moz-txt-link-freetext=22 href=3D=22http=3A//lists=2Eovirt=
=2Eorg/mailman/listinfo/users=22=3Ehttp=3A//lists=2Eovirt=2Eorg/mailman/lis=
tinfo/users=3C/a=3E
=3C/pre=3E
=3C/blockquote=3E
=3Cbr=3E
=20=
=3CBR /=3E
=3CBR /=3E
=3Cb style=3D=22color=3A=23604c78=22=3E=3C/b=3E=3Cbr=3E=3Cspan style=3D=22c=
olor=3A=23604c78=3B=22=3E=3Cfont color=3D=22000000=22=3E=3Cspan style=3D=22=
mso-fareast-language=3Aen-gb=3B=22 lang=3D=22NL=22=3EMet vriendelijke groet=
=2C With kind regards=2C=3Cbr=3E=3Cbr=3E=3C/span=3EJorick Astrego=3C/font=
=3E=3C/span=3E=3Cb style=3D=22color=3A=23604c78=22=3E=3Cbr=3E=3Cbr=3ENetbul=
ae Virtualization Experts =3C/b=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3B=
border-top=3A1px solid =23ccc=3B=22=3E=3Ctable style=3D=22width=3A 522px=22=
=3E=3Ctbody=3E=3Ctr=3E=3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=
=22=3ETel=3A 053 20 30 270=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bf=
ont-size=3A 10px=22=3Einfo=40netbulae=2Eeu=3C/td=3E =3Ctd style=3D=22wid=
th=3A 130px=3Bfont-size=3A 10px=22=3EStaalsteden 4-3A=3C/td=3E =3Ctd sty=
le=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EKvK 08198180=3C/td=3E=3C/tr=
=3E=3Ctr=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EFax=
=3A 053 20 30 271=3C/td=3E =3Ctd style=3D=22width=3A 130px=3Bfont-size=
=3A 10px=22=3Ewww=2Enetbulae=2Eeu=3C/td=3E =3Ctd style=3D=22width=3A 130=
px=3Bfont-size=3A 10px=22=3E7547 TA Enschede=3C/td=3E =3Ctd style=3D=22w=
idth=3A 130px=3Bfont-size=3A 10px=22=3EBTW NL821234584B01=3C/td=3E=3C/tr=3E=
=3C/tbody=3E=3C/table=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3Bborder-top=
=3A1px solid =23ccc=3B=22=3E=3CBR /=3E
=3C/body=3E
=3C/html=3E
--------------030308050700080206070001--
8:03 p.m.
New subject: VMs freezing during heals
--Apple-Mail=_328EDD08-7E1B-45BB-8C4D-3B30BE5DF33F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=windows-1252
I hadn=92t revisited it yet, but it is possible to use cgroups to limit =
glusterfs=92s cpu usage, might help you out.
Andrew Wklau has a blog post about it: =
http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroup=
s/
Careful about how far you throttle it down, if it=92s your VMs disk it=92s=
rebuilding, you=92ll pause it anyway I=92d expect.
On Apr 4, 2015, at 8:57 AM, Jorick Astrego
<j.astrego(a)netbulae.eu> =
wrote:
=20
=20
=20
On 04/03/2015 10:04 PM, Alastair Neil wrote:
> Any follow up on this?
>=20
> Are there known issues using a replica 3 glsuter datastore with lvm =
thin
provisioned bricks?
>=20
> On 20 March 2015 at 15:22, Alastair Neil <ajneil.tech(a)gmail.com =
<mailto:ajneil.tech@gmail.com>> wrote:
> CentOS 6.6
> =20
> vdsm-4.16.10-8.gitc937927.el6
> glusterfs-3.6.2-1.el6
> 2.6.32 - 504.8.1.el6.x86_64
>=20
> moved to 3.6 specifically to get the snapshotting feature, hence my =
desire to
migrate to thinly provisioned lvm bricks.
=20
=20
Well on the glusterfs mailinglist there have been discussions:
=20
=20
> 3.6.2 is a major release and introduces some new features in cluster =
wide
concept. Additionally it is not stable yet.
=20
=20
=20
=20
>=20
>=20
> On 20 March 2015 at 14:57, Darrell Budic <budic(a)onholyground.com =
<mailto:budic@onholyground.com>> wrote:
> What version of gluster are you running on these?
>=20
> I=92ve seen high load during heals bounce my hosted engine around due =
to
overall system load, but never pause anything else. Cent 7 combo =
storage/host systems, gluster 3.5.2.
>=20
>=20
>> On Mar 20, 2015, at 9:57 AM, Alastair Neil <ajneil.tech(a)gmail.com =
<mailto:ajneil.tech@gmail.com>> wrote:
>>=20
>> Pranith
>>=20
>> I have run a pretty straightforward test. I created a two brick 50 =
G
replica volume with normal lvm bricks, and installed two servers, one =
centos 6.6 and one centos 7.0. I kicked off bonnie++ on both to =
generate some file system activity and then made the volume replica 3. =
I saw no issues on the servers. =20
>>=20
>> Not clear if this is a sufficiently rigorous test and the Volume I =
have
had issues on is a 3TB volume with about 2TB used.
>>=20
>> -Alastair
>>=20
>>=20
>> On 19 March 2015 at 12:30, Alastair Neil <ajneil.tech(a)gmail.com =
<mailto:ajneil.tech@gmail.com>> wrote:
>> I don't think I have the resources to test it
meaningfully. I have =
about 50 vms on my primary storage domain. I might be able
to set up a =
small 50 GB volume and provision 2 or 3 vms running test loads but I'm =
not sure it would be comparable. I'll give it a try and let you know if =
I see similar behaviour.
>>=20
>> On 19 March 2015 at 11:34, Pranith Kumar Karampuri =
<pkarampu(a)redhat.com <mailto:pkarampu@redhat.com>> wrote:
>> Without thinly provisioned lvm.
>>=20
>> Pranith
>>=20
>> On 03/19/2015 08:01 PM, Alastair Neil wrote:
>>> do you mean raw partitions as bricks or simply with out thin =
provisioned lvm?
>>>=20
>>>=20
>>>=20
>>> On 19 March 2015 at 00:32, Pranith Kumar Karampuri =
<pkarampu(a)redhat.com <mailto:pkarampu@redhat.com>> wrote:
>>> Could you let me know if you see this problem without lvm
as well?
>>>=20
>>> Pranith
>>>=20
>>> On 03/18/2015 08:25 PM, Alastair Neil wrote:
>>>> I am in the process of replacing the bricks with thinly =
provisioned lvs yes.
>>>>=20
>>>>=20
>>>>=20
>>>> On 18 March 2015 at 09:35, Pranith Kumar Karampuri =
<pkarampu(a)redhat.com <mailto:pkarampu@redhat.com>> wrote:
>>>> hi,
>>>> Are you using thin-lvm based backend on which the bricks are =
created?
>>>>=20
>>>> Pranith
>>>>=20
>>>> On 03/18/2015 02:05 AM, Alastair Neil wrote:
>>>>> I have a Ovirt cluster with 6 VM hosts and 4 gluster nodes. There =
are two virtualisation clusters one with two nehelem nodes and one with =
four sandybridge nodes. My master storage domain is a GlusterFS backed =
by a replica 3 gluster volume from 3 of the gluster nodes. The engine =
is a hosted engine 3.5.1 on 3 of the sandybridge nodes, with storage =
broviede by nfs from a different gluster volume. All the hosts are =
CentOS 6.6.
>>>>>=20
>>>>> vdsm-4.16.10-8.gitc937927.el6
>>>>> glusterfs-3.6.2-1.el6
>>>>> 2.6.32 - 504.8.1.el6.x86_64
>>>>>=20
>>>>> Problems happen when I try to add a new brick or replace a brick =
eventually the self heal will kill the VMs. In the VM's logs I see =
kernel hung task messages.=20
>>>>>=20
>>>>> Mar 12 23:05:16 static1 kernel: INFO: task nginx:1736 blocked for =
more than 120 seconds.
>>>>> Mar 12 23:05:16 static1 kernel: Not tainted
=
2.6.32-504.3.3.el6.x86_64 #1
>>>>> Mar 12 23:05:16 static1 kernel: "echo 0 >
=
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> Mar 12 23:05:16 static1 kernel: nginx D
0000000000000001 =
0 1736 1735 0x00000080
>>>>> Mar 12 23:05:16 static1 kernel: ffff8800778b17a8
0000000000000082 =
0000000000000000 00000000000126c0
>>>>> Mar 12 23:05:16 static1 kernel: ffff88007e5c6500
ffff880037170080 =
0006ce5c85bd9185 ffff88007e5c64d0
>>>>> Mar 12 23:05:16 static1 kernel: ffff88007a614ae0
00000001722b64ba =
ffff88007a615098 ffff8800778b1fd8
>>>>> Mar 12 23:05:16 static1 kernel: Call Trace:
>>>>> Mar 12 23:05:16 static1 kernel: [<ffffffff8152a885>] =
schedule_timeout+0x215/0x2e0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8152a503>] =
wait_for_common+0x123/0x180
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81064b90>] ? =
default_wake_function+0x0/0x20
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0210a76>] ? =
_xfs_buf_read+0x46/0x60 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa02063c7>] ? =
xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8152a61d>] =
wait_for_completion+0x1d/0x20
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa020ff5b>] =
xfs_buf_iowait+0x9b/0x100 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa02063c7>] ? =
xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0210a76>] =
_xfs_buf_read+0x46/0x60 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0210b3b>] =
xfs_buf_read+0xab/0x100 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa02063c7>] =
xfs_trans_read_buf+0x197/0x410 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01ee6a4>] =
xfs_imap_to_bp+0x54/0x130 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01f077b>] =
xfs_iread+0x7b/0x1b0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811ab77e>] ? =
inode_init_always+0x11e/0x1c0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01eb5ee>] =
xfs_iget+0x27e/0x6e0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01eae1d>] ? =
xfs_iunlock+0x5d/0xd0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0209366>] =
xfs_lookup+0xc6/0x110 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0216024>] =
xfs_vn_lookup+0x54/0xa0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119dc65>] =
do_lookup+0x1a5/0x230
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119e8f4>] =
__link_path_walk+0x7a4/0x1000
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811738e7>] ? =
cache_grow+0x217/0x320
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119f40a>] =
path_walk+0x6a/0xe0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8119f61b>] =
filename_lookup+0x6b/0xc0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811a0747>] =
user_path_at+0x57/0xa0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa0204e74>] ? =
_xfs_trans_commit+0x214/0x2a0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffffa01eae3e>] ? =
xfs_iunlock+0x7e/0xd0 [xfs]
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81193bc0>] =
vfs_fstatat+0x50/0xa0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff811aaf5d>] ? =
touch_atime+0x14d/0x1a0
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81193d3b>] =
vfs_stat+0x1b/0x20
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff81193d64>] =
sys_newstat+0x24/0x50
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff810e5c87>] ? =
audit_syscall_entry+0x1d7/0x200
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff810e5a7e>] ? =
__audit_syscall_exit+0x25e/0x290
>>>>> Mar 12 23:05:16 static1 kernel:
[<ffffffff8100b072>] =
system_call_fastpath+0x16/0x1b
>>>>>=20
>>>>>=20
>>>>> I am wondering if my volume settings are causing this. Can =
anyone with more knowledge take a look and let me know:
>>>>>=20
>>>>> network.remote-dio: on
>>>>> performance.stat-prefetch: off
>>>>> performance.io-cache: off
>>>>> performance.read-ahead: off
>>>>> performance.quick-read: off
>>>>> nfs.export-volumes: on
>>>>> network.ping-timeout: 20
>>>>> cluster.self-heal-readdir-size: 64KB
>>>>> cluster.quorum-type: auto
>>>>> cluster.data-self-heal-algorithm: diff
>>>>> cluster.self-heal-window-size: 8
>>>>> cluster.heal-timeout: 500
>>>>> cluster.self-heal-daemon: on
>>>>> cluster.entry-self-heal: on
>>>>> cluster.data-self-heal: on
>>>>> cluster.metadata-self-heal: on
>>>>> cluster.readdir-optimize: on
>>>>> cluster.background-self-heal-count: 20
>>>>> cluster.rebalance-stats: on
>>>>> cluster.min-free-disk: 5%
>>>>> cluster.eager-lock: enable
>>>>> storage.owner-uid: 36
>>>>> storage.owner-gid: 36
>>>>> auth.allow:*
>>>>> user.cifs: disable
>>>>> cluster.server-quorum-ratio: 51%
>>>>>=20
>>>>> Many Thanks, Alastair
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>>>>=20
>>>>=20
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>>>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>>>>=20
>>>>=20
>>>=20
>>>=20
>>=20
>>=20
>>=20
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
>=20
>=20
>=20
=20
=20
>=20
>=20
>=20
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
=20
=20
Met vriendelijke groet, With kind regards,
=20
Jorick Astrego
=20
Netbulae Virtualization Experts=20
Tel: 053 20 30 270 info(a)netbulae.eu Staalsteden 4-3A =
KvK 08198180
Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW =
NL821234584B01
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--Apple-Mail=_328EDD08-7E1B-45BB-8C4D-3B30BE5DF33F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=windows-1252
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dwindows-1252"></head><body style=3D"word-wrap: break-word;
=
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">I hadn=92t revisited it yet, but it is possible to use =
cgroups to limit glusterfs=92s cpu usage, might help you out.<div =
class=3D""><br class=3D""></div><div
class=3D"">Andrew Wklau has a blog =
post about it: <a =
href=3D"http://www.andrewklau.com/controlling-glusterfsd-cpu-outbrea...
h-cgroups/" =
class=3D"">http://www.andrewklau.com/controlling-glusterfsd-...
with-cgroups/</a></div><div class=3D""><br
class=3D""></div><div =
class=3D"">Careful about how far you throttle it down, if it=92s your =
VMs disk it=92s rebuilding, you=92ll pause it anyway I=92d =
expect.</div><div class=3D""><br
class=3D""><div><blockquote type=3D"cite"=
class=3D""><div class=3D"">On Apr 4, 2015, at 8:57 AM,
Jorick Astrego =
<<a href=3D"mailto:j.astrego@netbulae.eu" =
class=3D"">j.astrego(a)netbulae.eu</a>&gt; wrote:</div><br
=
class=3D"Apple-interchange-newline"><div class=3D"">
=20
<meta content=3D"text/html; charset=3Dwindows-1252" =
http-equiv=3D"Content-Type" class=3D"">
=20
<div bgcolor=3D"#FFFFFF" text=3D"#000000"
class=3D"">
<br class=3D"">
<br class=3D"">
<div class=3D"moz-cite-prefix">On 04/03/2015 10:04 PM, Alastair Neil
wrote:<br class=3D"">
</div>
<blockquote =
cite=3D"mid:CA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w@mail.gmail=
.com" type=3D"cite" class=3D"">
<div dir=3D"ltr" class=3D"">Any follow up on this?
<div class=3D""><br class=3D"">
</div>
<div class=3D""> Are there known issues using a replica 3
=
glsuter datastore
with lvm thin provisioned bricks?</div>
</div>
<div class=3D"gmail_extra"><br class=3D"">
<div class=3D"gmail_quote">On 20 March 2015 at 15:22, Alastair
Neil <span dir=3D"ltr" class=3D""><<a =
moz-do-not-send=3D"true" href=3D"mailto:ajneil.tech@gmail.com" =
target=3D"_blank"
class=3D"">ajneil.tech(a)gmail.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr" class=3D"">
<div class=3D"">CentOS 6.6</div>
<span class=3D"">
<div class=3D""> </div>
<blockquote class=3D"gmail_quote" =
style=3D"font-size:13px;margin:0px 0px 0px
=
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"> vdsm-4.16.10-8.gitc937927.el6<br =
class=3D"">
glusterfs-3.6.2-1.el6<br class=3D"">
2.6.32 - 504.8.1.el6.x86_64</blockquote>
<div class=3D""><br class=3D"">
</div>
</span>
<div class=3D"">moved to 3.6 specifically to get the =
snapshotting
feature, hence my desire to migrate to thinly
provisioned lvm bricks.</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class=3D"">
<br class=3D"">
Well on the glusterfs mailinglist there have been discussions:<br =
class=3D"">
<br class=3D"">
<br class=3D"">
<blockquote type=3D"cite" class=3D"">3.6.2 is a major
release and =
introduces some
new features in cluster wide concept. Additionally it is not
stable yet.</blockquote>
<br class=3D"">
<br class=3D"">
<br class=3D"">
<br class=3D"">
<blockquote =
cite=3D"mid:CA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w@mail.gmail=
.com" type=3D"cite" class=3D"">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr" class=3D"">
<div class=3D""><br class=3D"">
</div>
<div class=3D"gmail_extra"><br class=3D"">
<div class=3D"gmail_quote">On 20 March 2015 at 14:57,
Darrell Budic <span dir=3D"ltr"
class=3D""><<a =
moz-do-not-send=3D"true" href=3D"mailto:budic@onholyground.com" =
target=3D"_blank"
class=3D"">budic(a)onholyground.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0
=
0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style=3D"word-wrap:break-word"
class=3D"">What =
version of
gluster are you running on these?
<div class=3D""><br class=3D"">
</div>
<div class=3D"">I=92ve seen high load during heals
=
bounce my
hosted engine around due to overall system load,
but never pause anything else. Cent 7 combo
storage/host systems, gluster 3.5.2.</div>
<div class=3D"">
<div class=3D"">
<div class=3D""><br class=3D"">
</div>
<div class=3D""><br class=3D"">
<div class=3D"">
<blockquote type=3D"cite"
class=3D"">
<div class=3D"">On Mar 20, 2015, at 9:57
=
AM,
Alastair Neil <<a =
moz-do-not-send=3D"true" href=3D"mailto:ajneil.tech@gmail.com" =
target=3D"_blank"
class=3D"">ajneil.tech(a)gmail.com</a>&gt;
wrote:</div>
<br class=3D"">
<div class=3D"">
<div dir=3D"ltr"
class=3D"">Pranith
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">I have run a pretty
straightforward test. I =
created a
two brick 50 G replica volume with
normal lvm bricks, and installed
two servers, one centos 6.6 and
one centos 7.0. I kicked off
bonnie++ on both to generate some
file system activity and then made
the volume replica 3. I saw =
no
issues on the servers. =
</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">Not clear if this is
=
a
sufficiently rigorous test and the
Volume I have had issues on is a
3TB volume with about 2TB =
used.</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"">-Alastair</div>
<div class=3D""><br
class=3D"">
</div>
<div class=3D"gmail_extra"><br =
class=3D"">
<div class=3D"gmail_quote">On 19
March 2015 at 12:30, Alastair
Neil <span dir=3D"ltr" =
class=3D""><<a moz-do-not-send=3D"true" =
href=3D"mailto:ajneil.tech@gmail.com" target=3D"_blank" =
class=3D"">ajneil.tech(a)gmail.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote class=3D"gmail_quote" =
style=3D"margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir=3D"ltr"
class=3D"">I =
don't think I
have the resources to test
it meaningfully. I =
have
about 50 vms on my primary
storage domain. I =
might be
able to set up a small 50 GB
volume and provision 2 or 3
vms running test loads but
I'm not sure it would be
comparable. I'll give =
it a
try and let you know if I
see similar behaviour.</div>
<div class=3D"">
<div class=3D"">
<div =
class=3D"gmail_extra"><br class=3D"">
<div =
class=3D"gmail_quote">On
19 March 2015 at
11:34, Pranith Kumar
Karampuri <span =
dir=3D"ltr" class=3D""><<a
moz-do-not-send=3D"true" =
href=3D"mailto:pkarampu@redhat.com" target=3D"_blank" =
class=3D"">pkarampu(a)redhat.com</a>&gt;</span>
wrote:<br class=3D"">
<blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px
#ccc
=
solid;padding-left:1ex">
<div text=3D"#000000" =
bgcolor=3D"#FFFFFF" class=3D"">
Without thinly
provisioned =
lvm.<span class=3D""><font color=3D"#888888"
class=3D""><br class=3D"">
<br class=3D"">
=
Pranith</font></span>
<div class=3D"">
<div =
class=3D""><br class=3D"">
<div =
class=3D"">On
03/19/2015
08:01 PM,
Alastair Neil
wrote:<br =
class=3D"">
</div>
<blockquote =
type=3D"cite" class=3D"">
<div dir=3D"ltr"=
class=3D"">do
you mean raw
partitions as
bricks or
simply with
out thin
provisioned
lvm?
<div =
class=3D""><br class=3D"">
</div>
<div =
class=3D""><br class=3D"">
</div>
</div>
<div =
class=3D"gmail_extra"><br class=3D"">
<div =
class=3D"gmail_quote">On
19 March 2015
at 00:32,
Pranith Kumar
Karampuri =
<span dir=3D"ltr" class=3D""><<a
moz-do-not-send=3D"true" =
href=3D"mailto:pkarampu@redhat.com" target=3D"_blank" =
class=3D"">pkarampu(a)redhat.com</a>&gt;</span>
wrote:<br =
class=3D"">
<blockquote =
class=3D"gmail_quote" style=3D"margin:0
0 0
=
.8ex;border-left:1px
#ccc
=
solid;padding-left:1ex">
<div =
text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D""> Could
you let me know =
if you see this problem without
lvm as =
well?<span class=3D""><font color=3D"#888888"
class=3D""><br class=3D"">
<br class=3D"">
=
Pranith</font></span>
<div
class=3D"">=
<div =
class=3D""><br class=3D"">
<div =
class=3D"">On
03/18/2015
08:25 PM,
Alastair Neil
wrote:<br =
class=3D"">
</div>
<blockquote =
type=3D"cite" class=3D"">
<div dir=3D"ltr"=
class=3D"">I
am in the
process of
replacing the
bricks with
thinly
provisioned
lvs yes.
<div =
class=3D""><br class=3D"">
</div>
<div =
class=3D""><br class=3D"">
</div>
</div>
<div =
class=3D"gmail_extra"><br class=3D"">
<div =
class=3D"gmail_quote">On
18 March 2015
at 09:35,
Pranith Kumar
Karampuri =
<span dir=3D"ltr" class=3D""><<a
moz-do-not-send=3D"true" =
href=3D"mailto:pkarampu@redhat.com" target=3D"_blank" =
class=3D"">pkarampu(a)redhat.com</a>&gt;</span>
wrote:<br =
class=3D"">
<blockquote =
class=3D"gmail_quote" style=3D"margin:0
0 0
=
.8ex;border-left:1px
#ccc
=
solid;padding-left:1ex">
<div =
text=3D"#000000" bgcolor=3D"#FFFFFF" class=3D"">
hi,<br class=3D"">
=
Are you
using thin-lvm
based backend
on which the
bricks are
created?<br =
class=3D"">
<br class=3D"">
Pranith
<div
class=3D"">=
<div =
class=3D""><br class=3D"">
<div =
class=3D"">On
03/18/2015
02:05 AM,
Alastair Neil
wrote:<br =
class=3D"">
</div>
</div>
</div>
<blockquote =
type=3D"cite" class=3D"">
<div
class=3D"">=
<div
class=3D"">=
<div dir=3D"ltr"=
class=3D"">I
have a Ovirt
cluster with 6
VM hosts and 4
gluster nodes.
There are two
virtualisation
clusters one
with two
nehelem nodes
and one with
four
=
sandybridge
nodes. My
master storage
domain is a
GlusterFS
backed by a
replica 3
gluster volume
from 3 of the
gluster
nodes. =
The
engine is a
hosted engine
3.5.1 on 3 of
the
sandybridge
nodes, with
storage
broviede by
nfs from a
different
gluster
volume. =
All
the hosts are
CentOS 6.6.
<div =
class=3D""><br class=3D"">
</div>
<blockquote =
class=3D"gmail_quote" style=3D"margin:0px
0px 0px
=
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"> vdsm-4.16.10-8.gitc937927.el6<br =
class=3D"">
glusterfs-3.6.2-1.el6<br class=3D"">
2.6.32 -
=
504.8.1.el6.x86_64</blockquote>
<div =
class=3D""><br class=3D"">
</div>
<div =
class=3D"">Problems
happen when I
try to add a
new brick or
replace a
brick
eventually the
self heal will
kill the VMs.
In the VM's
logs I see
kernel hung
task
=
messages. </div>
<div =
class=3D""><br class=3D"">
</div>
<div
class=3D"">=
<blockquote =
class=3D"gmail_quote" style=3D"margin:0px
0px 0px
=
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"><font face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel: INFO:
task
nginx:1736
blocked for
more than 120
seconds.<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel: =
Not =
tainted
=
2.6.32-504.3.3.el6.x86_64
#1<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel: "echo
0 >
=
/proc/sys/kernel/hung_task_timeout_secs"
disables this
message.<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel: nginx
=
D
=
0000000000000001
=
0 1736
1735
0x00000080<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
ffff8800778b17a8
=
0000000000000082
=
0000000000000000
00000000000126c0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
ffff88007e5c6500
=
ffff880037170080
=
0006ce5c85bd9185
ffff88007e5c64d0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
ffff88007a614ae0
=
00000001722b64ba
=
ffff88007a615098
ffff8800778b1fd8<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel: Call
Trace:<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8152a885>]
schedule_timeout+0x215/0x2e0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8152a503>]
wait_for_common+0x123/0x180<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff81064b90>]
?
=
default_wake_function+0x0/0x20<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa0210a76>]
?
=
_xfs_buf_read+0x46/0x60
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa02063c7>]
?
=
xfs_trans_read_buf+0x197/0x410
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8152a61d>]
wait_for_completion+0x1d/0x20<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa020ff5b>]
=
xfs_buf_iowait+0x9b/0x100
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa02063c7>]
?
=
xfs_trans_read_buf+0x197/0x410
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa0210a76>]
=
_xfs_buf_read+0x46/0x60
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa0210b3b>]
=
xfs_buf_read+0xab/0x100
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa02063c7>]
=
xfs_trans_read_buf+0x197/0x410
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa01ee6a4>]
=
xfs_imap_to_bp+0x54/0x130
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa01f077b>]
=
xfs_iread+0x7b/0x1b0
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff811ab77e>]
?
=
inode_init_always+0x11e/0x1c0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa01eb5ee>]
=
xfs_iget+0x27e/0x6e0
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa01eae1d>]
?
=
xfs_iunlock+0x5d/0xd0
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa0209366>]
=
xfs_lookup+0xc6/0x110
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa0216024>]
=
xfs_vn_lookup+0x54/0xa0
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8119dc65>]
do_lookup+0x1a5/0x230<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8119e8f4>]
__link_path_walk+0x7a4/0x1000<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff811738e7>]
?
=
cache_grow+0x217/0x320<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8119f40a>]
path_walk+0x6a/0xe0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8119f61b>]
filename_lookup+0x6b/0xc0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff811a0747>]
user_path_at+0x57/0xa0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa0204e74>]
?
=
_xfs_trans_commit+0x214/0x2a0
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffffa01eae3e>]
?
=
xfs_iunlock+0x7e/0xd0
[xfs]<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff81193bc0>]
vfs_fstatat+0x50/0xa0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff811aaf5d>]
?
=
touch_atime+0x14d/0x1a0<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff81193d3b>]
vfs_stat+0x1b/0x20<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff81193d64>]
sys_newstat+0x24/0x50<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff810e5c87>]
?
=
audit_syscall_entry+0x1d7/0x200<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff810e5a7e>]
?
=
__audit_syscall_exit+0x25e/0x290<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">Mar
12 23:05:16
static1
kernel:
=
[<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b</font></blockquote>
</div>
<div =
class=3D""><br class=3D"">
</div>
<div =
class=3D""><br class=3D"">
</div>
<div =
class=3D"">I am
wondering if
my volume
settings are
causing =
this.
Can anyone
with more
knowledge take
a look and let
me know:</div>
<div =
class=3D""><br class=3D"">
</div>
<div
class=3D"">=
<blockquote =
class=3D"gmail_quote" style=3D"margin:0px
0px 0px
=
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"><font face=3D"monospace,
monospace" =
class=3D"">network.remote-dio:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">performance.stat-prefetch:
off<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">performance.io-cache:
off<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">performance.read-ahead:
off<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">performance.quick-read:
off<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">nfs.export-volumes:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">network.ping-timeout:
20<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.self-heal-readdir-size:
64KB<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.quorum-type:
auto<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.data-self-heal-algorithm:
diff<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.self-heal-window-size:
8<br
class=3D"">=
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.heal-timeout:
500<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.self-heal-daemon:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.entry-self-heal:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.data-self-heal:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.metadata-self-heal:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.readdir-optimize:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.background-self-heal-count:
20<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.rebalance-stats:
on<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.min-free-disk:
5%<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.eager-lock:
enable<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">storage.owner-uid:
36<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">storage.owner-gid:
36<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">auth.allow:*<br class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">user.cifs:
disable<br =
class=3D"">
</font><font =
face=3D"monospace,
monospace" =
class=3D"">cluster.server-quorum-ratio:
=
51%</font></blockquote>
</div>
<div =
class=3D""><br class=3D"">
</div>
<div =
class=3D"">Many
Thanks,
=
Alastair</div>
<div =
class=3D""><br class=3D"">
</div>
</div>
<br class=3D"">
<fieldset =
class=3D""></fieldset>
<br class=3D"">
</div>
</div>
<pre =
class=3D"">_______________________________________________
Users mailing list
<a moz-do-not-send=3D"true" href=3D"mailto:Users@ovirt.org" =
target=3D"_blank" class=3D"">Users(a)ovirt.org</a>
<a moz-do-not-send=3D"true" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<...
</pre>
</blockquote>
<br class=3D"">
</div>
<br class=3D"">
_______________________________________________<br class=3D"">
Users mailing
list<br =
class=3D"">
<a =
moz-do-not-send=3D"true" href=3D"mailto:Users@ovirt.org"
target=3D"_blank"=
class=3D"">Users(a)ovirt.org</a><br class=3D"">
<a =
moz-do-not-send=3D"true" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... =
class=3D"">
<br class=3D"">
</blockquote>
</div>
<br class=3D"">
</div>
</blockquote>
<br class=3D"">
</div>
</div>
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</blockquote>
<br class=3D"">
</div>
</div>
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</div>
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</div>
_______________________________________________<br class=3D"">
Users mailing list<br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"mailto:Users@ovirt.org" target=3D"_blank" =
class=3D"">Users(a)ovirt.org</a><br class=3D"">
<a moz-do-not-send=3D"true" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
target=3D"_blank" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<... =
class=3D"">
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class=3D"">
</div>
</div>
<div class=3D"HOEnZb">
<div class=3D"h5"><br class=3D"">
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br class=3D"">
<br class=3D"">
<blockquote =
cite=3D"mid:CA+SarwqNuvVGUDDjhDRbNii-foMGAyaVibxyMGM5AEPzRkDu+w@mail.gmail=
.com" type=3D"cite" class=3D"">
<div class=3D"gmail_extra">
<div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=3D"HOEnZb">
<div class=3D"h5">
</div>
</div>
</blockquote>
</div>
<br class=3D"">
</div>
<br class=3D"">
<fieldset class=3D"mimeAttachmentHeader"></fieldset>
<br class=3D"">
<pre wrap=3D"" =
class=3D"">_______________________________________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated" =
href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class=3D"moz-txt-link-freetext" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br class=3D"">
<br class=3D"">
<br class=3D"">
<b style=3D"color:#604c78" class=3D""></b><br
class=3D""><span =
style=3D"color:#604c78;" class=3D""><span =
style=3D"mso-fareast-language:en-gb;" lang=3D"NL"
class=3D"">Met =
vriendelijke groet, With kind regards,<br class=3D""><br =
class=3D""></span>Jorick Astrego</span><b
style=3D"color:#604c78" =
class=3D""><br class=3D""><br
class=3D"">Netbulae Virtualization Experts =
</b><br class=3D""><hr style=3D"border:none;border-top:1px
solid #ccc;" =
class=3D""><table style=3D"width: 522px"
class=3D""><tbody class=3D""><tr =
class=3D""><td style=3D"width: 130px;font-size: 10px"
class=3D"">Tel: =
053 20 30 270</td> <td style=3D"width: 130px;font-size: 10px" =
class=3D""><a href=3D"mailto:info@netbulae.eu" =
class=3D"">info(a)netbulae.eu</a></td> <td
style=3D"width: =
130px;font-size: 10px" class=3D"">Staalsteden 4-3A</td> <td
=
style=3D"width: 130px;font-size: 10px" class=3D"">KvK =
08198180</td></tr><tr class=3D""> <td
style=3D"width: =
130px;font-size: 10px" class=3D"">Fax: 053 20 30 271</td>
<td =
style=3D"width: 130px;font-size: 10px" class=3D""><a =
href=3D"http://www.netbulae.eu"
class=3D"">www.netbulae.eu</a></td> =
<td style=3D"width: 130px;font-size: 10px" class=3D"">7547 TA =
Enschede</td> <td style=3D"width: 130px;font-size: 10px"
class=3D"">BTW=
NL821234584B01</td></tr></tbody></table><br
class=3D""><hr =
style=3D"border:none;border-top:1px solid #ccc;" class=3D""><br
=
class=3D"">
</div>
_______________________________________________<br class=3D"">Users =
mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org"
=
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br =
class=3D""></div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_328EDD08-7E1B-45BB-8C4D-3B30BE5DF33F--
3517
days inactive
3534
days old
5 comments
3 participants
participants (3)
-
Alastair Neil -
Darrell Budic -
Jorick Astrego