[ovirt-users] Huge Glsuter Issues - oVirt 4.1.7

Fri Nov 24 14:11:42 UTC 2017

Hallo Kasturi,

I have the issues only when a backup job is running.
I have not tested it with fuse or gfapi directly.

What would be a good test for fuse? Just DD to the mounted gluster storage?

How can I test it with gfapi?

Thanks for your support

Kind regards,

Florian Nolden

Head of IT at Xilloc Medical B.V.

———————————————————————————————

Disclaimer: The content of this e-mail, including any attachments, are
confidential and are intended for the sole use of the individual or entity
to which it is addressed. If you have received it by mistake please let us
know by reply and then delete it from your system. Any distribution,
copying or dissemination of this message is expected to conform to all
legal stipulations governing the use of information.

2017-11-24 9:40 GMT+01:00 Kasturi Narra <knarra at redhat.com>:

> Hi Florian,
>
>        Are you seeing these issues with gfapi or fuse access as well ?
>
> Thanks
> kasturi
>
> On Fri, Nov 24, 2017 at 3:06 AM, Florian Nolden <f.nolden at xilloc.com>
> wrote:
>
>> I have the same issue when I run backup tasks during the night.
>>
>> I have a Gluster setup with a 1TB SSD on each of the tree nodes. Maybe
>> its related to bug: https://bugzilla.redhat.com/show_bug.cgi?id=1430847
>>
>> sanlock.log:
>> 2017-11-23 00:46:42 3410597 [1114]: s15 check_our_lease warning 60
>> last_success 3410537
>> 2017-11-23 00:46:43 3410598 [1114]: s15 check_our_lease warning 61
>> last_success 3410537
>> 2017-11-23 00:46:44 3410599 [1114]: s15 check_our_lease warning 62
>> last_success 3410537
>> 2017-11-23 00:46:45 3410600 [1114]: s15 check_our_lease warning 63
>> last_success 3410537
>> 2017-11-23 00:46:46 3410601 [1114]: s15 check_our_lease warning 64
>> last_success 3410537
>> 2017-11-23 00:46:47 3410602 [1114]: s15 check_our_lease warning 65
>> last_success 3410537
>> 2017-11-23 00:46:48 3410603 [1114]: s15 check_our_lease warning 66
>> last_success 3410537
>> 2017-11-23 00:46:49 3410603 [28384]: s15 delta_renew long write time 46
>> sec
>> 2017-11-23 00:46:49 3410603 [28384]: s15 renewed 3410557 delta_length 46
>> too long
>> 2017-11-23 02:48:04 3417878 [28384]: s15 delta_renew long write time 10
>> sec
>> 2017-11-23 02:57:23 3418438 [28384]: s15 delta_renew long write time 34
>> sec
>> 2017-11-23 02:57:23 3418438 [28384]: s15 renewed 3418404 delta_length 34
>> too long
>>
>>
>> vdsm.log | grep "WARN"
>> 017-11-23 00:20:05,544+0100 WARN  (jsonrpc/0) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=63.7199999997) (vm:5109)
>> 2017-11-23 00:20:06,840+0100 WARN  (check/loop) [storage.check] Checker
>> u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae
>> -1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 10.00
>> seconds (check:279)
>> 2017-11-23 00:20:13,853+0100 WARN  (periodic/170)
>> [virt.periodic.VmDispatcher] could not run <class
>> 'vdsm.virt.periodic.UpdateVolumes'> on [u'e1f26ea9-9294-4d9c-8f70-d59f96dec5f7']
>> (periodic:308)
>> 2017-11-23 00:20:15,031+0100 WARN  (jsonrpc/2) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=73.21) (vm:5109)
>> 2017-11-23 00:20:20,586+0100 WARN  (jsonrpc/4) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=78.7599999998) (vm:5109)
>> 2017-11-23 00:21:06,849+0100 WARN  (check/loop) [storage.check] Checker
>> u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae
>> -1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 10.01
>> seconds (check:279)
>> 2017-11-23 00:21:13,847+0100 WARN  (periodic/167)
>> [virt.periodic.VmDispatcher] could not run <class
>> 'vdsm.virt.periodic.UpdateVolumes'> on [u'd8f22423-9fe3-4c06-97dc-5c9e9f5b33c8']
>> (periodic:308)
>> 2017-11-23 00:22:13,854+0100 WARN  (periodic/172)
>> [virt.periodic.VmDispatcher] could not run <class
>> 'vdsm.virt.periodic.UpdateVolumes'> on [u'd8f22423-9fe3-4c06-97dc-5c9e9f5b33c8']
>> (periodic:308)
>> 2017-11-23 00:22:16,846+0100 WARN  (check/loop) [storage.check] Checker
>> u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae
>> -1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 9.99
>> seconds (check:279)
>> 2017-11-23 00:23:06,040+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=64.2199999997) (vm:5109)
>> 2017-11-23 00:23:06,850+0100 WARN  (check/loop) [storage.check] Checker
>> u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae
>> -1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 9.98
>> seconds (check:279)
>> 2017-11-23 00:23:13,845+0100 WARN  (periodic/169)
>> [virt.periodic.VmDispatcher] could not run <class
>> 'vdsm.virt.periodic.UpdateVolumes'> on [u'5ef506de-44b9-4ced-9b7f-b90ee098f4f7']
>> (periodic:308)
>> 2017-11-23 00:23:16,855+0100 WARN  (jsonrpc/7) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=75.0300000003 <030%20000%200003>)
>> (vm:5109)
>> 2017-11-23 00:23:21,082+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=79.2599999998) (vm:5109)
>> 2017-11-23 00:25:31,488+0100 WARN  (libvirt/events) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') unknown eventid 8 args
>> ('/rhev/data-center/00000001-0001-0001-0001-000000000370/f0e
>> 21aae-1237-4dd3-88ec-81254d29c372/images/1a1b9620-52fc-4008-
>> 9047-15cd725f8bd8/90b
>> 913ba-e03f-46c5-bccf-bae011fcdd55', 4, 3, 8) (clientIF:549)
>> 2017-11-23 00:25:32,372+0100 WARN  (libvirt/events) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') unknown eventid 8 args
>> ('/rhev/data-center/00000001-0001-0001-0001-000000000370/f0e
>> 21aae-1237-4dd3-88ec-81254d29c372/images/1a1b9620-52fc-4008-
>> 9047-15cd725f8bd8/90b
>> 913ba-e03f-46c5-bccf-bae011fcdd55', 4, 0, 8) (clientIF:549)
>> 2017-11-23 00:45:56,851+0100 WARN  (check/loop) [storage.check] Checker
>> u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae
>> -1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 10.00
>> seconds (check:279)
>> 2017-11-23 00:46:13,850+0100 WARN  (periodic/172)
>> [virt.periodic.VmDispatcher] could not run <class
>> 'vdsm.virt.periodic.UpdateVolumes'> on [u'e1f26ea9-9294-4d9c-8f70-d59f96dec5f7',
>> u'5ef506de-44b9-4ced-9b7f-b90ee098f4f7'] (periodic:308)
>> 2017-11-23 00:46:36,013+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became
>> unresponsive (command timeout, age=64.0899999999) (vm:5109)
>> 2017-11-23 00:46:38,805+0100 WARN  (jsonrpc/2) [virt.vm]
>> (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became
>> unresponsive (command timeout, age=66.8799999999) (vm:5109)
>> 2017-11-23 00:46:40,439+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='930ecaca-ef2f-490a-a4df-e4f0dad218aa') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,440+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,441+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,442+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,442+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='0cf9b0cb-7c53-4bab-b879-0bdf190b293c') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,443+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,444+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,445+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='d8f22423-9fe3-4c06-97dc-5c9e9f5b33c8') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,446+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='ea36f7bd-1790-4b42-b7e1-6d8e2ef0487b') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:40,446+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='82ed235e-37bb-4d67-8db9-61d39340f951') monitor became
>> unresponsive (command timeout, age=68.5199999996) (vm:5109)
>> 2017-11-23 00:46:46,116+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='930ecaca-ef2f-490a-a4df-e4f0dad218aa') monitor became
>> unresponsive (command timeout, age=74.1899999995) (vm:5109)
>> 2017-11-23 00:46:46,118+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became
>> unresponsive (command timeout, age=74.1899999995) (vm:5109)
>> 2017-11-23 00:46:46,119+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,120+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,121+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='0cf9b0cb-7c53-4bab-b879-0bdf190b293c') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,123+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,124+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,125+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='d8f22423-9fe3-4c06-97dc-5c9e9f5b33c8') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,127+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='ea36f7bd-1790-4b42-b7e1-6d8e2ef0487b') monitor became
>> unresponsive (command timeout, age=74.1999999993) (vm:5109)
>> 2017-11-23 00:46:46,128+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='82ed235e-37bb-4d67-8db9-61d39340f951') monitor became
>> unresponsive (command timeout, age=74.21) (vm:5109)
>> 2017-11-23 00:46:46,509+0100 WARN  (jsonrpc/3) [virt.vm]
>> (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became
>> unresponsive (command timeout, age=74.5899999999) (vm:5109)
>> 2017-11-23 00:46:48,187+0100 WARN  (jsonrpc/7) [virt.vm]
>> (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became
>> unresponsive (command timeout, age=76.2599999998) (vm:5109)
>> 2017-11-23 00:46:49,825+0100 WARN  (periodic/173)
>> [virt.sampling.StatsCache] dropped stale old sample: sampled 7705208.650000
>> stored 7705268.650000 (sampling:442)
>> 2017-11-23 00:46:49,835+0100 WARN  (periodic/176)
>> [virt.sampling.StatsCache] dropped stale old sample: sampled 7705253.650000
>> stored 7705268.650000 (sampling:442)
>> 2017-11-23 00:46:49,854+0100 WARN  (periodic/171)
>> [virt.sampling.StatsCache] dropped stale old sample: sampled 7705238.650000
>> stored 7705268.650000 (sampling:442)
>> 2017-11-23 00:46:49,866+0100 WARN  (periodic/174)
>> [virt.sampling.StatsCache] dropped stale old sample: sampled 7705223.650000
>> stored 7705268.650000 (sampling:442)
>> 2017-11-23 00:46:55,488+0100 WARN  (jsonrpc/0) [virt.vm]
>> (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became
>> unresponsive (command timeout, age=83.5699999994) (vm:5109)
>> 2017-11-23 00:46:55,488+0100 WARN  (jsonrpc/0) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=83.5699999994) (vm:5109)
>> 2017-11-23 00:46:55,489+0100 WARN  (jsonrpc/0) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=83.5699999994) (vm:5109)
>> 2017-11-23 00:46:55,491+0100 WARN  (jsonrpc/0) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=83.5699999994) (vm:5109)
>> 2017-11-23 00:47:01,742+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became
>> unresponsive (command timeout, age=89.8199999994) (vm:5109)
>> 2017-11-23 00:47:01,743+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=89.8199999994) (vm:5109)
>> 2017-11-23 00:47:01,744+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=89.8199999994) (vm:5109)
>> 2017-11-23 00:47:01,746+0100 WARN  (jsonrpc/1) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=89.8199999994) (vm:5109)
>> 2017-11-23 00:47:10,531+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=98.6099999994) (vm:5109)
>> 2017-11-23 00:47:10,532+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=98.6099999994) (vm:5109)
>> 2017-11-23 00:47:10,534+0100 WARN  (jsonrpc/6) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=98.6099999994) (vm:5109)
>> 2017-11-23 00:47:16,950+0100 WARN  (jsonrpc/7) [virt.vm]
>> (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became
>> unresponsive (command timeout, age=105.029999999) (vm:5109)
>> 2017-11-23 00:47:16,951+0100 WARN  (jsonrpc/7) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=105.029999999) (vm:5109)
>> 2017-11-23 00:47:16,953+0100 WARN  (jsonrpc/7) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=105.029999999) (vm:5109)
>> 2017-11-23 00:47:25,578+0100 WARN  (jsonrpc/4) [virt.vm]
>> (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became
>> unresponsive (command timeout, age=113.659999999) (vm:5109)
>> 2017-11-23 00:47:25,581+0100 WARN  (jsonrpc/4) [virt.vm]
>> (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became
>> unresponsive (command timeout, age=113.659999999) (vm:5109)
>>
>> Kind regards,
>>
>>
>> Florian Nolden
>>
>> Head of IT at Xilloc Medical B.V.
>>
>> ———————————————————————————————
>>
>> Disclaimer: The content of this e-mail, including any attachments, are
>> confidential and are intended for the sole use of the individual or entity
>> to which it is addressed. If you have received it by mistake please let us
>> know by reply and then delete it from your system. Any distribution,
>> copying or dissemination of this message is expected to conform to all
>> legal stipulations governing the use of information.
>>
>> 2017-11-23 11:25 GMT+01:00 Sven Achtelik <Sven.Achtelik at eps.aero>:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I’m experiencing huge issues when working with big VMs on Gluster
>>> volumes. Doing a Snapshot or removing a big Disk lead to the effect that
>>> the SPM node is getting non responsive. Fencing is than kicking in and
>>> taking the node down with the hard reset/reboot.
>>>
>>>
>>>
>>> My setup has three nodes with 10Gbit/s NICs for the Gluster network. The
>>> Bricks are on Raid-6 with a 1GB cache on the raid controller and the
>>> volumes are setup as follows:
>>>
>>>
>>>
>>> Volume Name: data
>>>
>>> Type: Replicate
>>>
>>> Volume ID: c734d678-91e3-449c-8a24-d26b73bef965
>>>
>>> Status: Started
>>>
>>> Snapshot Count: 0
>>>
>>> Number of Bricks: 1 x 3 = 3
>>>
>>> Transport-type: tcp
>>>
>>> Bricks:
>>>
>>> Brick1: ovirt-node01-gfs.storage.lan:/gluster/brick2/data
>>>
>>> Brick2: ovirt-node02-gfs.storage.lan:/gluster/brick2/data
>>>
>>> Brick3: ovirt-node03-gfs.storage.lan:/gluster/brick2/data
>>>
>>> Options Reconfigured:
>>>
>>> features.barrier: disable
>>>
>>> cluster.granular-entry-heal: enable
>>>
>>> performance.readdir-ahead: on
>>>
>>> performance.quick-read: off
>>>
>>> performance.read-ahead: off
>>>
>>> performance.io-cache: off
>>>
>>> performance.stat-prefetch: on
>>>
>>> cluster.eager-lock: enable
>>>
>>> network.remote-dio: off
>>>
>>> cluster.quorum-type: auto
>>>
>>> cluster.server-quorum-type: server
>>>
>>> storage.owner-uid: 36
>>>
>>> storage.owner-gid: 36
>>>
>>> features.shard: on
>>>
>>> features.shard-block-size: 512MB
>>>
>>> performance.low-prio-threads: 32
>>>
>>> cluster.data-self-heal-algorithm: full
>>>
>>> cluster.locking-scheme: granular
>>>
>>> cluster.shd-wait-qlength: 10000
>>>
>>> cluster.shd-max-threads: 6
>>>
>>> network.ping-timeout: 30
>>>
>>> user.cifs: off
>>>
>>> nfs.disable: on
>>>
>>> performance.strict-o-direct: on
>>>
>>> server.event-threads: 4
>>>
>>> client.event-threads: 4
>>>
>>>
>>>
>>> It feel like the System looks up during snapshotting or removing of a
>>> big disk and this delay triggers things to go wrong. Is there anything that
>>> is not setup right on my gluster or is this behavior normal with bigger
>>> disks (50GB+) ? Is there a reliable option for caching with SSDs ?
>>>
>>>
>>>
>>> Thank you,
>>>
>>> Sven
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20171124/8fa93b4b/attachment.html>