Hi, since some time we get Error Messages from Sanlock, and so far i was not able to figure out what exactly they try to tell and more important if its something which can be ignored or needs to be fixed (and how).

Here are the Versions we are using currently: Engine ovirt-engine-3.5.6.2-1.el6.noarch Nodes vdsm-4.16.34-0.el7.centos.x86_64 sanlock-3.2.4-1.el7.x86_64 libvirt-lock-sanlock-1.2.17-13.el7_2.3.x86_64 libvirt-daemon-1.2.17-13.el7_2.3.x86_64 libvirt-lock-sanlock-1.2.17-13.el7_2.3.x86_64 libvirt-1.2.17-13.el7_2.3.x86_64 -- snip -- May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109 [60137]: verify_leader 2 wrong space name 4643f652-8014-4951-8a1a-02af41e67d08 f757b127-a951-4fa9-bf90-81180c0702e6 /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109 [60137]: leader1 delta_acquire_begin error -226 lockspace f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109 [60137]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0 May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109 [60137]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0 May 30 09:55:27 vm2 sanlock[1094]: 2016-05-30 09:55:27+0200 294109 [60137]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033 May 30 09:55:28 vm2 sanlock[1094]: 2016-05-30 09:55:28+0200 294110 [1099]: s9703 add_lockspace fail result -226 May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140 [60331]: verify_leader 2 wrong space name 4643f652-8014-4951-8a1a-02af41e67d08 f757b127-a951-4fa9-bf90-81180c0702e6 /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140 [60331]: leader1 delta_acquire_begin error -226 lockspace f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140 [60331]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0 May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140 [60331]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0 May 30 09:55:58 vm2 sanlock[1094]: 2016-05-30 09:55:58+0200 294140 [60331]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033 May 30 09:55:59 vm2 sanlock[1094]: 2016-05-30 09:55:59+0200 294141 [1098]: s9704 add_lockspace fail result -226 May 30 09:56:05 vm2 sanlock[1094]: 2016-05-30 09:56:05+0200 294148 [1094]: s1527 check_other_lease invalid for host 0 0 ts 7566376 name in 4643f652-8014-4951-8a1a-02af41e67d08 May 30 09:56:05 vm2 sanlock[1094]: 2016-05-30 09:56:05+0200 294148 [1094]: s1527 check_other_lease leader 12212010 owner 1 11 ts 7566376 sn f757b127-a951-4fa9-bf90-81180c0702e6 rn f888524b-27aa-4724-8bae-051f9e950a21.vm1.intern May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170 [60496]: verify_leader 2 wrong space name 4643f652-8014-4951-8a1a-02af41e67d08 f757b127-a951-4fa9-bf90-81180c0702e6 /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170 [60496]: leader1 delta_acquire_begin error -226 lockspace f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170 [60496]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0 May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170 [60496]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0 May 30 09:56:28 vm2 sanlock[1094]: 2016-05-30 09:56:28+0200 294170 [60496]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033 May 30 09:56:29 vm2 sanlock[1094]: 2016-05-30 09:56:29+0200 294171 [6415]: s9705 add_lockspace fail result -226 May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200 [60645]: verify_leader 2 wrong space name 4643f652-8014-4951-8a1a-02af41e67d08 f757b127-a951-4fa9-bf90-81180c0702e6 /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200 [60645]: leader1 delta_acquire_begin error -226 lockspace f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200 [60645]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0 May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200 [60645]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0 May 30 09:56:58 vm2 sanlock[1094]: 2016-05-30 09:56:58+0200 294200 [60645]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033 May 30 09:56:59 vm2 sanlock[1094]: 2016-05-30 09:56:59+0200 294201 [6373]: s9706 add_lockspace fail result -226 May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230 [60806]: verify_leader 2 wrong space name 4643f652-8014-4951-8a1a-02af41e67d08 f757b127-a951-4fa9-bf90-81180c0702e6 /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230 [60806]: leader1 delta_acquire_begin error -226 lockspace f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230 [60806]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0 May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230 [60806]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0 May 30 09:57:28 vm2 sanlock[1094]: 2016-05-30 09:57:28+0200 294230 [60806]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033 May 30 09:57:29 vm2 sanlock[1094]: 2016-05-30 09:57:29+0200 294231 [6399]: s9707 add_lockspace fail result -226 May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260 [60946]: verify_leader 2 wrong space name 4643f652-8014-4951-8a1a-02af41e67d08 f757b127-a951-4fa9-bf90-81180c0702e6 /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260 [60946]: leader1 delta_acquire_begin error -226 lockspace f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260 [60946]: leader2 path /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids offset 0 May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260 [60946]: leader3 m 12212010 v 30003 ss 512 nh 0 mh 1 oi 2 og 8 lv 0 May 30 09:57:58 vm2 sanlock[1094]: 2016-05-30 09:57:58+0200 294260 [60946]: leader4 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern ts 3786679 cs 1474f033 May 30 09:57:59 vm2 sanlock[1094]: 2016-05-30 09:57:59+0200 294261 [6352]: s9708 add_lockspace fail result -226 -- snip -- sanlock log_dump also shows errors -- snip -- 2016-05-30 09:53:23+0200 7526415 [1017]: s567 check_other_lease invalid for host 0 0 ts 7566376 name in 4643f652-8014-4951-8a1a-02af41e67d08 2016-05-30 09:53:23+0200 7526415 [1017]: s567 check_other_lease leader 12212010 owner 1 11 ts 7566376 sn f757b127-a951-4fa9-bf90-81180c0702e6 rn f888524b-27aa-4724-8bae-051f9e950a21.vm1.intern 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid for host 0 0 ts 3786679 name in f757b127-a951-4fa9-bf90-81180c0702e6 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader 12212010 owner 2 8 ts 3786679 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 1eed8aa9-8fb5-4d27-8d1c-03ebce2c36d4.vm2.intern 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid for host 0 0 ts 6622415 name in f757b127-a951-4fa9-bf90-81180c0702e6 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader 12212010 owner 3 14 ts 6622415 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 51c8f8e2-f9d8-462c-866c-e4052213ea81.vm3.intern 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid for host 0 0 ts 6697413 name in f757b127-a951-4fa9-bf90-81180c0702e6 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader 12212010 owner 4 4 ts 6697413 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 8d4f32dc-f595-4254-bfcb-d96e5057e110.vm4.intern 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid for host 0 0 ts 7563413 name in f757b127-a951-4fa9-bf90-81180c0702e6 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader 12212010 owner 5 8 ts 7563413 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn 9fe564e6-cf40-4403-9fc0-eb118e1ee2cf.vm5.intern 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease invalid for host 0 0 ts 6129706 name in f757b127-a951-4fa9-bf90-81180c0702e6 2016-05-30 09:53:33+0200 7526425 [1017]: s568 check_other_lease leader 12212010 owner 6 169 ts 6129706 sn 4643f652-8014-4951-8a1a-02af41e67d08 rn b99ff588-f1ad-43f2-bd6c-6869f54d424d.vm1-tiny.i 2016-05-30 09:53:47+0200 7526439 [14576]: cmd_read_resource 23,58 /dev/c2212f15-35b7-4fa0-b13c-b50befda0af9/leases:1048576 -- snip -- any hint whould be highly appreciated Juergen _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

> verify_leader 2 wrong space name > 4643f652-8014-4951-8a1a-02af41e67d08 > f757b127-a951-4fa9-bf90-81180c0702e6 > /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids > leader1 delta_acquire_begin error -226 lockspace > f757b127-a951-4fa9-bf90-81180c0702e6 host_id 2 VDSM has tried to join VG/lockspace/storage-domain "f757b127" on LV /dev/f757b127-a951-4fa9-bf90-81180c0702e6/ids. But sanlock finds that lockspace "4643f652" is initialized on that storage, i.e. inconsistency between the leases formatted on disk and what the leases are being used for. That should never happen unless sanlock and/or storage are used/moved/copied wrongly. The error is a sanlock sanity check to catch misuse. > s1527 check_other_lease invalid for host 0 0 ts 7566376 name in > 4643f652-8014-4951-8a1a-02af41e67d08 > s1527 check_other_lease leader 12212010 owner 1 11 ts 7566376 > sn f757b127-a951-4fa9-bf90-81180c0702e6 rn f888524b-27aa-4724-8bae-051f9e950a21.vm1.intern Apparently sanlock is already managing a lockspace called "4643f652" when it finds another lease in that lockspace has the inconsistent/corrupt name "f757b127". I can't say what steps might have been done to lead to this. This is a mess that's been caused by improper use of storage, and various sanity checks in sanlock have all reported errors for "impossible" conditions indicating that something catastrophic has been done to the storage it's using. Some fundamental rules are not being followed.

On Thu, Jun 02, 2016 at 06:47:37PM +0300, Nir Soffer wrote: >> This is a mess that's been caused by improper use of storage, and various >> sanity checks in sanlock have all reported errors for "impossible" >> conditions indicating that something catastrophic has been done to the >> storage it's using. Some fundamental rules are not being followed. > > Thanks David. > > Do you need more output from sanlock to understand this issue? I can think of nothing more to learn from sanlock. I'd suggest tighter, higher level checking or control of storage. Low level sanity checks detecting lease corruption are not a convenient place to work from.

What if we move all vm off the lun which causes this error, drop the lun and recreated it. Will we "migrate" the error with the VM to a different lun or could this be a fix?

Am 6/3/2016 um 10:08 AM schrieb InterNetX - Juergen Gotteswinter: > Hello David, > > thanks for your explanation of those messages, is there any possibility > to get rid of this? i already figured out that it might be an corruption > of the ids file, but i didnt find anything about re-creating or other > solutions to fix this. > > Imho this occoured after an outage where several hosts, and the iscsi > SAN has been fenced and/or rebooted. > > Thanks, > > Juergen > > > Am 6/2/2016 um 6:03 PM schrieb David Teigland: >> On Thu, Jun 02, 2016 at 06:47:37PM +0300, Nir Soffer wrote: >>>> This is a mess that's been caused by improper use of storage, and various >>>> sanity checks in sanlock have all reported errors for "impossible" >>>> conditions indicating that something catastrophic has been done to the >>>> storage it's using. Some fundamental rules are not being followed. >>> >>> Thanks David. >>> >>> Do you need more output from sanlock to understand this issue? >> >> I can think of nothing more to learn from sanlock. I'd suggest tighter, >> higher level checking or control of storage. Low level sanity checks >> detecting lease corruption are not a convenient place to work from. >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >