[ovirt-users] Huge Glsuter Issues - oVirt 4.1.7

Thu Nov 23 21:36:20 UTC 2017

I have the same issue when I run backup tasks during the night.

I have a Gluster setup with a 1TB SSD on each of the tree nodes. Maybe its
related to bug: https://bugzilla.redhat.com/show_bug.cgi?id=1430847

sanlock.log:
2017-11-23 00:46:42 3410597 [1114]: s15 check_our_lease warning 60
last_success 3410537
2017-11-23 00:46:43 3410598 [1114]: s15 check_our_lease warning 61
last_success 3410537
2017-11-23 00:46:44 3410599 [1114]: s15 check_our_lease warning 62
last_success 3410537
2017-11-23 00:46:45 3410600 [1114]: s15 check_our_lease warning 63
last_success 3410537
2017-11-23 00:46:46 3410601 [1114]: s15 check_our_lease warning 64
last_success 3410537
2017-11-23 00:46:47 3410602 [1114]: s15 check_our_lease warning 65
last_success 3410537
2017-11-23 00:46:48 3410603 [1114]: s15 check_our_lease warning 66
last_success 3410537
2017-11-23 00:46:49 3410603 [28384]: s15 delta_renew long write time 46 sec
2017-11-23 00:46:49 3410603 [28384]: s15 renewed 3410557 delta_length 46
too long
2017-11-23 02:48:04 3417878 [28384]: s15 delta_renew long write time 10 sec
2017-11-23 02:57:23 3418438 [28384]: s15 delta_renew long write time 34 sec
2017-11-23 02:57:23 3418438 [28384]: s15 renewed 3418404 delta_length 34
too long

vdsm.log | grep "WARN"
017-11-23 00:20:05,544+0100 WARN  (jsonrpc/0) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=63.7199999997) (vm:5109)
2017-11-23 00:20:06,840+0100 WARN  (check/loop) [storage.check] Checker
u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata'
is blocked for 10.00 seconds (check:279)
2017-11-23 00:20:13,853+0100 WARN  (periodic/170)
[virt.periodic.VmDispatcher] could not run <class
'vdsm.virt.periodic.UpdateVolumes'> on
[u'e1f26ea9-9294-4d9c-8f70-d59f96dec5f7'] (periodic:308)
2017-11-23 00:20:15,031+0100 WARN  (jsonrpc/2) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=73.21) (vm:5109)
2017-11-23 00:20:20,586+0100 WARN  (jsonrpc/4) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=78.7599999998) (vm:5109)
2017-11-23 00:21:06,849+0100 WARN  (check/loop) [storage.check] Checker
u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata'
is blocked for 10.01 seconds (check:279)
2017-11-23 00:21:13,847+0100 WARN  (periodic/167)
[virt.periodic.VmDispatcher] could not run <class
'vdsm.virt.periodic.UpdateVolumes'> on
[u'd8f22423-9fe3-4c06-97dc-5c9e9f5b33c8'] (periodic:308)
2017-11-23 00:22:13,854+0100 WARN  (periodic/172)
[virt.periodic.VmDispatcher] could not run <class
'vdsm.virt.periodic.UpdateVolumes'> on
[u'd8f22423-9fe3-4c06-97dc-5c9e9f5b33c8'] (periodic:308)
2017-11-23 00:22:16,846+0100 WARN  (check/loop) [storage.check] Checker
u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata'
is blocked for 9.99 seconds (check:279)
2017-11-23 00:23:06,040+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=64.2199999997) (vm:5109)
2017-11-23 00:23:06,850+0100 WARN  (check/loop) [storage.check] Checker
u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata'
is blocked for 9.98 seconds (check:279)
2017-11-23 00:23:13,845+0100 WARN  (periodic/169)
[virt.periodic.VmDispatcher] could not run <class
'vdsm.virt.periodic.UpdateVolumes'> on
[u'5ef506de-44b9-4ced-9b7f-b90ee098f4f7'] (periodic:308)
2017-11-23 00:23:16,855+0100 WARN  (jsonrpc/7) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=75.0300000003) (vm:5109)
2017-11-23 00:23:21,082+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=79.2599999998) (vm:5109)
2017-11-23 00:25:31,488+0100 WARN  (libvirt/events) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') unknown eventid 8 args
('/rhev/data-center/00000001-0001-0001-0001-000000000370/f0e21aae-1237-4dd3-88ec-81254d29c372/images/1a1b9620-52fc-4008-9047-15cd725f8bd8/90b
913ba-e03f-46c5-bccf-bae011fcdd55', 4, 3, 8) (clientIF:549)
2017-11-23 00:25:32,372+0100 WARN  (libvirt/events) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') unknown eventid 8 args
('/rhev/data-center/00000001-0001-0001-0001-000000000370/f0e21aae-1237-4dd3-88ec-81254d29c372/images/1a1b9620-52fc-4008-9047-15cd725f8bd8/90b
913ba-e03f-46c5-bccf-bae011fcdd55', 4, 0, 8) (clientIF:549)
2017-11-23 00:45:56,851+0100 WARN  (check/loop) [storage.check] Checker
u'/rhev/data-center/mnt/glusterSD/x-c01-n03:_fastIO/f0e21aae-1237-4dd3-88ec-81254d29c372/dom_md/metadata'
is blocked for 10.00 seconds (check:279)
2017-11-23 00:46:13,850+0100 WARN  (periodic/172)
[virt.periodic.VmDispatcher] could not run <class
'vdsm.virt.periodic.UpdateVolumes'> on
[u'e1f26ea9-9294-4d9c-8f70-d59f96dec5f7',
u'5ef506de-44b9-4ced-9b7f-b90ee098f4f7'] (periodic:308)
2017-11-23 00:46:36,013+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive
(command timeout, age=64.0899999999) (vm:5109)
2017-11-23 00:46:38,805+0100 WARN  (jsonrpc/2) [virt.vm]
(vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive
(command timeout, age=66.8799999999) (vm:5109)
2017-11-23 00:46:40,439+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='930ecaca-ef2f-490a-a4df-e4f0dad218aa') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,440+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,441+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,442+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,442+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='0cf9b0cb-7c53-4bab-b879-0bdf190b293c') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,443+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,444+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,445+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='d8f22423-9fe3-4c06-97dc-5c9e9f5b33c8') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,446+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='ea36f7bd-1790-4b42-b7e1-6d8e2ef0487b') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:40,446+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='82ed235e-37bb-4d67-8db9-61d39340f951') monitor became unresponsive
(command timeout, age=68.5199999996) (vm:5109)
2017-11-23 00:46:46,116+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='930ecaca-ef2f-490a-a4df-e4f0dad218aa') monitor became unresponsive
(command timeout, age=74.1899999995) (vm:5109)
2017-11-23 00:46:46,118+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive
(command timeout, age=74.1899999995) (vm:5109)
2017-11-23 00:46:46,119+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,120+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,121+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='0cf9b0cb-7c53-4bab-b879-0bdf190b293c') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,123+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,124+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,125+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='d8f22423-9fe3-4c06-97dc-5c9e9f5b33c8') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,127+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='ea36f7bd-1790-4b42-b7e1-6d8e2ef0487b') monitor became unresponsive
(command timeout, age=74.1999999993) (vm:5109)
2017-11-23 00:46:46,128+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='82ed235e-37bb-4d67-8db9-61d39340f951') monitor became unresponsive
(command timeout, age=74.21) (vm:5109)
2017-11-23 00:46:46,509+0100 WARN  (jsonrpc/3) [virt.vm]
(vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive
(command timeout, age=74.5899999999) (vm:5109)
2017-11-23 00:46:48,187+0100 WARN  (jsonrpc/7) [virt.vm]
(vmId='0bcf7520-3c60-42a1-8e6b-683af670e6cb') monitor became unresponsive
(command timeout, age=76.2599999998) (vm:5109)
2017-11-23 00:46:49,825+0100 WARN  (periodic/173)
[virt.sampling.StatsCache] dropped stale old sample: sampled 7705208.650000
stored 7705268.650000 (sampling:442)
2017-11-23 00:46:49,835+0100 WARN  (periodic/176)
[virt.sampling.StatsCache] dropped stale old sample: sampled 7705253.650000
stored 7705268.650000 (sampling:442)
2017-11-23 00:46:49,854+0100 WARN  (periodic/171)
[virt.sampling.StatsCache] dropped stale old sample: sampled 7705238.650000
stored 7705268.650000 (sampling:442)
2017-11-23 00:46:49,866+0100 WARN  (periodic/174)
[virt.sampling.StatsCache] dropped stale old sample: sampled 7705223.650000
stored 7705268.650000 (sampling:442)
2017-11-23 00:46:55,488+0100 WARN  (jsonrpc/0) [virt.vm]
(vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive
(command timeout, age=83.5699999994) (vm:5109)
2017-11-23 00:46:55,488+0100 WARN  (jsonrpc/0) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=83.5699999994) (vm:5109)
2017-11-23 00:46:55,489+0100 WARN  (jsonrpc/0) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=83.5699999994) (vm:5109)
2017-11-23 00:46:55,491+0100 WARN  (jsonrpc/0) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=83.5699999994) (vm:5109)
2017-11-23 00:47:01,742+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='e1f26ea9-9294-4d9c-8f70-d59f96dec5f7') monitor became unresponsive
(command timeout, age=89.8199999994) (vm:5109)
2017-11-23 00:47:01,743+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=89.8199999994) (vm:5109)
2017-11-23 00:47:01,744+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=89.8199999994) (vm:5109)
2017-11-23 00:47:01,746+0100 WARN  (jsonrpc/1) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=89.8199999994) (vm:5109)
2017-11-23 00:47:10,531+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=98.6099999994) (vm:5109)
2017-11-23 00:47:10,532+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=98.6099999994) (vm:5109)
2017-11-23 00:47:10,534+0100 WARN  (jsonrpc/6) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=98.6099999994) (vm:5109)
2017-11-23 00:47:16,950+0100 WARN  (jsonrpc/7) [virt.vm]
(vmId='0a83954f-56d1-42d0-88b9-825435055fd0') monitor became unresponsive
(command timeout, age=105.029999999) (vm:5109)
2017-11-23 00:47:16,951+0100 WARN  (jsonrpc/7) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=105.029999999) (vm:5109)
2017-11-23 00:47:16,953+0100 WARN  (jsonrpc/7) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=105.029999999) (vm:5109)
2017-11-23 00:47:25,578+0100 WARN  (jsonrpc/4) [virt.vm]
(vmId='245e104f-2bd5-4f77-81de-d75a593d77c5') monitor became unresponsive
(command timeout, age=113.659999999) (vm:5109)
2017-11-23 00:47:25,581+0100 WARN  (jsonrpc/4) [virt.vm]
(vmId='5ef506de-44b9-4ced-9b7f-b90ee098f4f7') monitor became unresponsive
(command timeout, age=113.659999999) (vm:5109)

Kind regards,

Florian Nolden

Head of IT at Xilloc Medical B.V.

———————————————————————————————

Disclaimer: The content of this e-mail, including any attachments, are
confidential and are intended for the sole use of the individual or entity
to which it is addressed. If you have received it by mistake please let us
know by reply and then delete it from your system. Any distribution,
copying or dissemination of this message is expected to conform to all
legal stipulations governing the use of information.

2017-11-23 11:25 GMT+01:00 Sven Achtelik <Sven.Achtelik at eps.aero>:

> Hi All,
>
>
>
> I’m experiencing huge issues when working with big VMs on Gluster volumes.
> Doing a Snapshot or removing a big Disk lead to the effect that the SPM
> node is getting non responsive. Fencing is than kicking in and taking the
> node down with the hard reset/reboot.
>
>
>
> My setup has three nodes with 10Gbit/s NICs for the Gluster network. The
> Bricks are on Raid-6 with a 1GB cache on the raid controller and the
> volumes are setup as follows:
>
>
>
> Volume Name: data
>
> Type: Replicate
>
> Volume ID: c734d678-91e3-449c-8a24-d26b73bef965
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 1 x 3 = 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: ovirt-node01-gfs.storage.lan:/gluster/brick2/data
>
> Brick2: ovirt-node02-gfs.storage.lan:/gluster/brick2/data
>
> Brick3: ovirt-node03-gfs.storage.lan:/gluster/brick2/data
>
> Options Reconfigured:
>
> features.barrier: disable
>
> cluster.granular-entry-heal: enable
>
> performance.readdir-ahead: on
>
> performance.quick-read: off
>
> performance.read-ahead: off
>
> performance.io-cache: off
>
> performance.stat-prefetch: on
>
> cluster.eager-lock: enable
>
> network.remote-dio: off
>
> cluster.quorum-type: auto
>
> cluster.server-quorum-type: server
>
> storage.owner-uid: 36
>
> storage.owner-gid: 36
>
> features.shard: on
>
> features.shard-block-size: 512MB
>
> performance.low-prio-threads: 32
>
> cluster.data-self-heal-algorithm: full
>
> cluster.locking-scheme: granular
>
> cluster.shd-wait-qlength: 10000
>
> cluster.shd-max-threads: 6
>
> network.ping-timeout: 30
>
> user.cifs: off
>
> nfs.disable: on
>
> performance.strict-o-direct: on
>
> server.event-threads: 4
>
> client.event-threads: 4
>
>
>
> It feel like the System looks up during snapshotting or removing of a big
> disk and this delay triggers things to go wrong. Is there anything that is
> not setup right on my gluster or is this behavior normal with bigger disks
> (50GB+) ? Is there a reliable option for caching with SSDs ?
>
>
>
> Thank you,
>
> Sven
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20171123/1bf85645/attachment.html>