[ovirt-users] Sanlock add Lockspace Errors

Mon May 30 04:06:41 EDT 2016

Hi,

since some time we get Error Messages from Sanlock, and so far i was not
able to figure out what exactly they try to tell and more important if
its something which can be ignored or needs to be fixed (and how).

Here are the Versions we are using currently:

Engine

ovirt-engine-3.5.6.2-1.el6.noarch

Nodes

vdsm-4.16.34-0.el7.centos.x86_64
sanlock-3.2.4-1.el7.x86_64
libvirt-lock-sanlock-1.2.17-13.el7_2.3.x86_64
libvirt-daemon-1.2.17-13.el7_2.3.x86_64
libvirt-lock-sanlock-1.2.17-13.el7_2.3.x86_64
libvirt-1.2.17-13.el7_2.3.x86_64

-- snip --
May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109
[60137]: verify_leader 2 wrong space name
4643f652-8014-4951-8a1a-02af41e67d08
f757b127-a951-4fa9-bf90-81180c0702e6
/dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids
May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109
[60137]: leader1 delta_acquire_begin error -226 lockspace
f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2
May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109
[60137]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0
May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109
[60137]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0
May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109
[60137]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033
May 30 09:55:28 vm2 sanlock[1094]: 2016-05-30 09:55:28+0200 294110
[1099]: s9703 add_lockspace fail result -226
May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140
[60331]: verify_leader 2 wrong space name
4643f652-8014-4951-8a1a-02af41e67d08
f757b127-a951-4fa9-bf90-81180c0702e6
/dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids
May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140
[60331]: leader1 delta_acquire_begin error -226 lockspace
f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2
May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140
[60331]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0
May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140
[60331]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0
May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140
[60331]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033
May 30 09:55:59 vm2 sanlock[1094]: 2016-05-30 09:55:59+0200 294141
[1098]: s9704 add_lockspace fail result -226
May 30 09:56:05 vm2 sanlock[1094]: 2016-05-30 09:56:05+0200 294148
[1094]: s1527 check_other_lease invalid for host 0 0 ts 7566376 name  in
4643f652-8014-4951-8a1a-02af41e67d08
May 30 09:56:05 vm2 sanlock[1094]: 2016-05-30 09:56:05+0200 294148
[1094]: s1527 check_other_lease leader 12212010 owner 1 11 ts 7566376 sn
f757b127-a951-4fa9-bf90-81180c0702e6 rn
f888524b-27aa-4724-8bae-051f9e950a21.vm1.intern
May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170
[60496]: verify_leader 2 wrong space name
4643f652-8014-4951-8a1a-02af41e67d08
f757b127-a951-4fa9-bf90-81180c0702e6
/dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids
May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170
[60496]: leader1 delta_acquire_begin error -226 lockspace
f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2
May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170
[60496]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0
May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170
[60496]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0
May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170
[60496]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033
May 30 09:56:29 vm2 sanlock[1094]: 2016-05-30 09:56:29+0200 294171
[6415]: s9705 add_lockspace fail result -226
May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200
[60645]: verify_leader 2 wrong space name
4643f652-8014-4951-8a1a-02af41e67d08
f757b127-a951-4fa9-bf90-81180c0702e6
/dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids
May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200
[60645]: leader1 delta_acquire_begin error -226 lockspace
f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2
May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200
[60645]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0
May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200
[60645]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0
May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200
[60645]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033
May 30 09:56:59 vm2 sanlock[1094]: 2016-05-30 09:56:59+0200 294201
[6373]: s9706 add_lockspace fail result -226
May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230
[60806]: verify_leader 2 wrong space name
4643f652-8014-4951-8a1a-02af41e67d08
f757b127-a951-4fa9-bf90-81180c0702e6
/dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids
May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230
[60806]: leader1 delta_acquire_begin error -226 lockspace
f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2
May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230
[60806]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0
May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230
[60806]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0
May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230
[60806]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033
May 30 09:57:29 vm2 sanlock[1094]: 2016-05-30 09:57:29+0200 294231
[6399]: s9707 add_lockspace fail result -226
May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260
[60946]: verify_leader 2 wrong space name
4643f652-8014-4951-8a1a-02af41e67d08
f757b127-a951-4fa9-bf90-81180c0702e6
/dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids
May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260
[60946]: leader1 delta_acquire_begin error -226 lockspace
f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2
May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260
[60946]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0
May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260
[60946]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0
May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260
[60946]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033
May 30 09:57:59 vm2 sanlock[1094]: 2016-05-30 09:57:59+0200 294261
[6352]: s9708 add_lockspace fail result -226
-- snip --

sanlock log_dump also shows errors

-- snip --

2016-05-30 09:53:23+0200 7526415 [1017]: s567 check_other_lease invalid
for host 0 0 ts 7566376 name  in 4643f652-8014-4951-8a1a-02af41e67d08
2016-05-30 09:53:23+0200 7526415 [1017]: s567 check_other_lease leader
12212010 owner 1 11 ts 7566376 sn f757b127-a951-4fa9-bf90-81180c0702e6
rn f888524b-27aa-4724-8bae-051f9e950a21.vm1.intern
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid
for host 0 0 ts 3786679 name  in f757b127-a951-4fa9-bf90-81180c0702e6
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader
12212010 owner 2 8 ts 3786679 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid
for host 0 0 ts 6622415 name  in f757b127-a951-4fa9-bf90-81180c0702e6
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader
12212010 owner 3 14 ts 6622415 sn 4643f652-8014-4951-8a1a-02af41e67d08
rn 51c8f8e2-f9d8-462c-866c-e4052213ea81.vm3.intern
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid
for host 0 0 ts 6697413 name  in f757b127-a951-4fa9-bf90-81180c0702e6
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader
12212010 owner 4 4 ts 6697413 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
8d4f32dc-f595-4254-bfcb-d96e5057e110.vm4.intern
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid
for host 0 0 ts 7563413 name  in f757b127-a951-4fa9-bf90-81180c0702e6
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader
12212010 owner 5 8 ts 7563413 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn
9fe564e6-cf40-4403-9fc0-eb118e1ee2cf.vm5.intern
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid
for host 0 0 ts 6129706 name  in f757b127-a951-4fa9-bf90-81180c0702e6
2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader
12212010 owner 6 169 ts 6129706 sn 4643f652-8014-4951-8a1a-02af41e67d08
rn b99ff588-f1ad-43f2-bd6c-6869f54d424d.vm1-tiny.i
2016-05-30 09:53:47+0200 7526439 [14576]: cmd_read_resource 23,58
/dev/c2212f15-35b7-4fa0-b13c-b50befda0af9/leases:1048576
-- snip --

any hint whould be highly appreciated

Juergen