Hi Florian,

       Are you seeing these issues with gfapi or fuse access as well ?

Thanks

kasturi

On Fri, Nov 24, 2017 at 3:06 AM, Florian Nolden <f.nolden@xilloc.com> wrote:

I have the same issue when I run backup tasks during the night.

I have a Gluster setup with a 1TB SSD on each of the tree nodes. Maybe its related to bug: https://bugzilla.redhat.com/show_bug.cgi?id=1430847

sanlock.log:

2017-11-23 00:46:42 3410597 [1114]: s15 check_our_lease warning 60 last_success 3410537

2017-11-23 00:46:43 3410598 [1114]: s15 check_our_lease warning 61 last_success 3410537

2017-11-23 00:46:44 3410599 [1114]: s15 check_our_lease warning 62 last_success 3410537

2017-11-23 00:46:45 3410600 [1114]: s15 check_our_lease warning 63 last_success 3410537

2017-11-23 00:46:46 3410601 [1114]: s15 check_our_lease warning 64 last_success 3410537

2017-11-23 00:46:47 3410602 [1114]: s15 check_our_lease warning 65 last_success 3410537

2017-11-23 00:46:48 3410603 [1114]: s15 check_our_lease warning 66 last_success 3410537

2017-11-23 00:46:49 3410603 [28384]: s15 delta_renew long write time 46 sec

2017-11-23 00:46:49 3410603 [28384]: s15 renewed 3410557 delta_length 46 too long

2017-11-23 02:48:04 3417878 [28384]: s15 delta_renew long write time 10 sec

2017-11-23 02:57:23 3418438 [28384]: s15 delta_renew long write time 34 sec

2017-11-23 02:57:23 3418438 [28384]: s15 renewed 3418404 delta_length 34 too long

vdsm.log | grep "WARN" 

017-11-23 00:20:05,544+0100 WARN  (jsonrpc/0) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=63.7199999997) (vm:5109)

2017-11-23 00:20:06,840+0100 WARN  (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 10.00 seconds (check:279)

2017-11-23 00:20:13,853+0100 WARN  (periodic/170) [virt.periodic.VmDispatcher] could not run <class 'vdsm.virt.periodic.UpdateVolumes'> on [u'e1f26ea9-9294-4d9c-8f70-d59f96dec5f7'] (periodic:308)

2017-11-23 00:20:15,031+0100 WARN  (jsonrpc/2) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=73.21) (vm:5109)

2017-11-23 00:20:20,586+0100 WARN  (jsonrpc/4) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=78.7599999998) (vm:5109)

2017-11-23 00:21:06,849+0100 WARN  (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 10.01 seconds (check:279)

2017-11-23 00:21:13,847+0100 WARN  (periodic/167) [virt.periodic.VmDispatcher] could not run <class 'vdsm.virt.periodic.UpdateVolumes'> on [u'd8f22423-9fe3-4c06-97dc-5c9e9f5b33c8'] (periodic:308)

2017-11-23 00:22:13,854+0100 WARN  (periodic/172) [virt.periodic.VmDispatcher] could not run <class 'vdsm.virt.periodic.UpdateVolumes'> on [u'd8f22423-9fe3-4c06-97dc-5c9e9f5b33c8'] (periodic:308)

2017-11-23 00:22:16,846+0100 WARN  (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 9.99 seconds (check:279)

2017-11-23 00:23:06,040+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=64.2199999997) (vm:5109)

2017-11-23 00:23:06,850+0100 WARN  (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 9.98 seconds (check:279)

2017-11-23 00:23:13,845+0100 WARN  (periodic/169) [virt.periodic.VmDispatcher] could not run <class 'vdsm.virt.periodic.UpdateVolumes'> on [u'5ef506de-44b9-4ced-9b7f-b90ee098f4f7'] (periodic:308)

2017-11-23 00:23:16,855+0100 WARN  (jsonrpc/7) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=75.0300000003) (vm:5109)

2017-11-23 00:23:21,082+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=79.2599999998) (vm:5109)

2017-11-23 00:25:31,488+0100 WARN  (libvirt/events) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') unknown eventid 8 args ('/rhev/data-center/00000001-0001-0001-0001-000000000370/f0e21aae-1237-4dd3-88ec-81254d29c372/images/1a1b9620-52fc-4008-9047-15cd725f8bd8/90b

913ba-e03f-46c5-bccf-bae011fcdd55', 4, 3, 8) (clientIF:549)

2017-11-23 00:25:32,372+0100 WARN  (libvirt/events) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') unknown eventid 8 args ('/rhev/data-center/00000001-0001-0001-0001-000000000370/f0e21aae-1237-4dd3-88ec-81254d29c372/images/1a1b9620-52fc-4008-9047-15cd725f8bd8/90b

913ba-e03f-46c5-bccf-bae011fcdd55', 4, 0, 8) (clientIF:549)

2017-11-23 00:45:56,851+0100 WARN  (check/loop) [storage.check] Checker u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata' is blocked for 10.00 seconds (check:279)

2017-11-23 00:46:13,850+0100 WARN  (periodic/172) [virt.periodic.VmDispatcher] could not run <class 'vdsm.virt.periodic.UpdateVolumes'> on [u'e1f26ea9-9294-4d9c-8f70-d59f96dec5f7', u'5ef506de-44b9-4ced-9b7f-b90ee098f4f7'] (periodic:308)

2017-11-23 00:46:36,013+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive (command timeout, age=64.0899999999) (vm:5109)

2017-11-23 00:46:38,805+0100 WARN  (jsonrpc/2) [virt.vm] (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive (command timeout, age=66.8799999999) (vm:5109)

2017-11-23 00:46:40,439+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='930ecaca-ef2f-490a-a4df-e4f0dad218aa') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,440+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,441+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,442+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,442+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='0cf9b0cb-7c53-4bab-b879-0bdf190b293c') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,443+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,444+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,445+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='d8f22423-9fe3-4c06-97dc-5c9e9f5b33c8') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,446+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='ea36f7bd-1790-4b42-b7e1-6d8e2ef0487b') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:40,446+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='82ed235e-37bb-4d67-8db9-61d39340f951') monitor became unresponsive (command timeout, age=68.5199999996) (vm:5109)

2017-11-23 00:46:46,116+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='930ecaca-ef2f-490a-a4df-e4f0dad218aa') monitor became unresponsive (command timeout, age=74.1899999995) (vm:5109)

2017-11-23 00:46:46,118+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive (command timeout, age=74.1899999995) (vm:5109)

2017-11-23 00:46:46,119+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,120+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,121+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='0cf9b0cb-7c53-4bab-b879-0bdf190b293c') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,123+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,124+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,125+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='d8f22423-9fe3-4c06-97dc-5c9e9f5b33c8') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,127+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='ea36f7bd-1790-4b42-b7e1-6d8e2ef0487b') monitor became unresponsive (command timeout, age=74.1999999993) (vm:5109)

2017-11-23 00:46:46,128+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='82ed235e-37bb-4d67-8db9-61d39340f951') monitor became unresponsive (command timeout, age=74.21) (vm:5109)

2017-11-23 00:46:46,509+0100 WARN  (jsonrpc/3) [virt.vm] (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive (command timeout, age=74.5899999999) (vm:5109)

2017-11-23 00:46:48,187+0100 WARN  (jsonrpc/7) [virt.vm] (vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive (command timeout, age=76.2599999998) (vm:5109)

2017-11-23 00:46:49,825+0100 WARN  (periodic/173) [virt.sampling.StatsCache] dropped stale old sample: sampled 7705208.650000 stored 7705268.650000 (sampling:442)

2017-11-23 00:46:49,835+0100 WARN  (periodic/176) [virt.sampling.StatsCache] dropped stale old sample: sampled 7705253.650000 stored 7705268.650000 (sampling:442)

2017-11-23 00:46:49,854+0100 WARN  (periodic/171) [virt.sampling.StatsCache] dropped stale old sample: sampled 7705238.650000 stored 7705268.650000 (sampling:442)

2017-11-23 00:46:49,866+0100 WARN  (periodic/174) [virt.sampling.StatsCache] dropped stale old sample: sampled 7705223.650000 stored 7705268.650000 (sampling:442)

2017-11-23 00:46:55,488+0100 WARN  (jsonrpc/0) [virt.vm] (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive (command timeout, age=83.5699999994) (vm:5109)

2017-11-23 00:46:55,488+0100 WARN  (jsonrpc/0) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=83.5699999994) (vm:5109)

2017-11-23 00:46:55,489+0100 WARN  (jsonrpc/0) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=83.5699999994) (vm:5109)

2017-11-23 00:46:55,491+0100 WARN  (jsonrpc/0) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=83.5699999994) (vm:5109)

2017-11-23 00:47:01,742+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive (command timeout, age=89.8199999994) (vm:5109)

2017-11-23 00:47:01,743+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=89.8199999994) (vm:5109)

2017-11-23 00:47:01,744+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=89.8199999994) (vm:5109)

2017-11-23 00:47:01,746+0100 WARN  (jsonrpc/1) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=89.8199999994) (vm:5109)

2017-11-23 00:47:10,531+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=98.6099999994) (vm:5109)

2017-11-23 00:47:10,532+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=98.6099999994) (vm:5109)

2017-11-23 00:47:10,534+0100 WARN  (jsonrpc/6) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=98.6099999994) (vm:5109)

2017-11-23 00:47:16,950+0100 WARN  (jsonrpc/7) [virt.vm] (vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive (command timeout, age=105.029999999) (vm:5109)

2017-11-23 00:47:16,951+0100 WARN  (jsonrpc/7) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=105.029999999) (vm:5109)

2017-11-23 00:47:16,953+0100 WARN  (jsonrpc/7) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=105.029999999) (vm:5109)

2017-11-23 00:47:25,578+0100 WARN  (jsonrpc/4) [virt.vm] (vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive (command timeout, age=113.659999999) (vm:5109)

2017-11-23 00:47:25,581+0100 WARN  (jsonrpc/4) [virt.vm] (vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive (command timeout, age=113.659999999) (vm:5109)

Kind regards,

Florian Nolden

Head of IT at Xilloc Medical B.V.

——————————————————————————————— 

Disclaimer: The content of this e-mail, including any attachments, are confidential and are intended for the sole use of the individual or entity to which it is addressed. If you have received it by mistake please let us know by reply and then delete it from your system. Any distribution, copying or dissemination of this message is expected to conform to all legal stipulations governing the use of information.

2017-11-23 11:25 GMT+01:00 Sven Achtelik <Sven.Achtelik@eps.aero>:

Hi All,

 

I’m experiencing huge issues when working with big VMs on Gluster volumes. Doing a Snapshot or removing a big Disk lead to the effect that the SPM node is getting non responsive. Fencing is than kicking in and taking the node down with the hard reset/reboot.

 

My setup has three nodes with 10Gbit/s NICs for the Gluster network. The Bricks are on Raid-6 with a 1GB cache on the raid controller and the volumes are setup as follows:

 

Volume Name: data

Type: Replicate

Volume ID: c734d678-91e3-449c-8a24-d26b73bef965

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: ovirt-node01-gfs.storage.lan:/gluster/brick2/data

Brick2: ovirt-node02-gfs.storage.lan:/gluster/brick2/data

Brick3: ovirt-node03-gfs.storage.lan:/gluster/brick2/data

Options Reconfigured:

features.barrier: disable

cluster.granular-entry-heal: enable

performance.readdir-ahead: on

performance.quick-read: off

performance.read-ahead: off

performance.io-cache: off

performance.stat-prefetch: on

cluster.eager-lock: enable

network.remote-dio: off

cluster.quorum-type: auto

cluster.server-quorum-type: server

storage.owner-uid: 36

storage.owner-gid: 36

features.shard: on

features.shard-block-size: 512MB

performance.low-prio-threads: 32

cluster.data-self-heal-algorithm: full

cluster.locking-scheme: granular

cluster.shd-wait-qlength: 10000

cluster.shd-max-threads: 6

network.ping-timeout: 30

user.cifs: off

nfs.disable: on

performance.strict-o-direct: on

server.event-threads: 4

client.event-threads: 4

 

It feel like the System looks up during snapshotting or removing of a big disk and this delay triggers things to go wrong. Is there anything that is not setup right on my gluster or is this behavior normal with bigger disks (50GB+) ? Is there a reliable option for caching with SSDs ?

 

Thank you,

Sven

_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users