Hello, 

I have a small 2-node cluster setup running Glusterfs in replication mode :

CentOS v6.5 
kernel-2.6.32-431.17.1.el6.x86_64
vdsm-4.14.6-0.el6.x86_64
ovirt-engine-3.4.0-1.el6.noarch  (on 1 node)

Basically I was running ovirt-engine v 3.3 for months fine and then upgraded to latest version of 3.3.X two days ago and could not join the nodes to the cluster due to a version mismatch,basically this:
https://www.mail-archive.com/users@ovirt.org/msg17241.html  . After trying to correct this problem I ended up upgrading to 3.4 which created a new and challenng problem for me. Every couple of hours I get error messages like this:

Jun  7 13:40:01 hv1 sanlock[2341]: 2014-06-07 13:40:01-0400 19647 [2341]: s3 check_our_lease warning 70 last_success 19577
Jun  7 13:40:02 hv1 sanlock[2341]: 2014-06-07 13:40:02-0400 19648 [2341]: s3 check_our_lease warning 71 last_success 19577
Jun  7 13:40:03 hv1 sanlock[2341]: 2014-06-07 13:40:03-0400 19649 [2341]: s3 check_our_lease warning 72 last_success 19577
Jun  7 13:40:04 hv1 sanlock[2341]: 2014-06-07 13:40:04-0400 19650 [2341]: s3 check_our_lease warning 73 last_success 19577
Jun  7 13:40:05 hv1 sanlock[2341]: 2014-06-07 13:40:05-0400 19651 [2341]: s3 check_our_lease warning 74 last_success 19577
Jun  7 13:40:06 hv1 sanlock[2341]: 2014-06-07 13:40:06-0400 19652 [2341]: s3 check_our_lease warning 75 last_success 19577
Jun  7 13:40:07 hv1 sanlock[2341]: 2014-06-07 13:40:07-0400 19653 [2341]: s3 check_our_lease warning 76 last_success 19577
Jun  7 13:40:08 hv1 sanlock[2341]: 2014-06-07 13:40:08-0400 19654 [2341]: s3 check_our_lease warning 77 last_success 19577
Jun  7 13:40:09 hv1 wdmd[2330]: test warning now 19654 ping 19644 close 0 renewal 19577 expire 19657 client 2341 sanlock_1e8615b0-7876-4a03-bdb0-352087fad0f3:1
Jun  7 13:40:09 hv1 wdmd[2330]: /dev/watchdog closed unclean
Jun  7 13:40:09 hv1 kernel: SoftDog: Unexpected close, not stopping watchdog!
Jun  7 13:40:09 hv1 sanlock[2341]: 2014-06-07 13:40:09-0400 19655 [2341]: s3 check_our_lease warning 78 last_success 19577
Jun  7 13:40:10 hv1 wdmd[2330]: test warning now 19655 ping 19644 close 19654 renewal 19577 expire 19657 client 2341 sanlock_1e8615b0-7876-4a03-bdb0-352087fad0f3:1
Jun  7 13:40:10 hv1 sanlock[2341]: 2014-06-07 13:40:10-0400 19656 [2341]: s3 check_our_lease warning 79 last_success 19577
Jun  7 13:40:11 hv1 wdmd[2330]: test warning now 19656 ping 19644 close 19654 renewal 19577 expire 19657 client 2341 sanlock_1e8615b0-7876-4a03-bdb0-352087fad0f3:1
Jun  7 13:40:11 hv1 sanlock[2341]: 2014-06-07 13:40:11-0400 19657 [2341]: s3 check_our_lease failed 80
Jun  7 13:40:11 hv1 sanlock[2341]: 2014-06-07 13:40:11-0400 19657 [2341]: s3 all pids clear

Jun  7 13:40:11 hv1 wdmd[2330]: /dev/watchdog reopen
Jun  7 13:41:32 hv1 sanlock[2341]: 2014-06-07 13:41:32-0400 19738 [5050]: s3 delta_renew write error -202
Jun  7 13:41:32 hv1 sanlock[2341]: 2014-06-07 13:41:32-0400 19738 [5050]: s3 renewal error -202 delta_length 140 last_success 19577
Jun  7 13:41:42 hv1 sanlock[2341]: 2014-06-07 13:41:42-0400 19748 [5050]: 1e8615b0 close_task_aio 0 0x7fd3040008c0 busy
Jun  7 13:41:52 hv1 sanlock[2341]: 2014-06-07 13:41:52-0400 19758 [5050]: 1e8615b0 close_task_aio 0 0x7fd3040008c0 busy

This makes one of the nodes not be able to see the storage and all its VMs will go into pause mode/stop. Wondering if you could provide some advice. Thank you

--Rizzo