[ovirt-users] sanlock ids file broken after server crash

[Maor Lipchuk]

On Sun, Jul 30, 2017 at 4:24 PM, Maor Lipchuk <mlipchuk at redhat.com> wrote:
> Hi David,
Sorry, I meant Johan

>
> I'm not sure how it got to that character in the first place.
> Nir, Is there a safe way to fix that while there are running VMs?
>
> Regards,
> Maor
>
> On Sun, Jul 30, 2017 at 11:58 AM, Johan Bernhardsson <johan at kafit.se> wrote:
>> (First reply did not get to the list)
>>
>> From sanlock.log:
>>
>> 2017-07-30 10:49:31+0200 1766275 [1171]: s310751 lockspace 0924ff77-
>> ef51-435b-b90d-50bfbf2e8de7:1:/rhev/data-
>> center/mnt/glusterSD/vbgsan02:_fs02/0924ff77-ef51-435b-b90d-
>> 50bfbf2e8de7/dom_md/ids:0
>> 2017-07-30 10:49:31+0200 1766275 [10496]: verify_leader 1 wrong space
>> name 0924ff77-ef51-435b-b90d-50bfbf2e<D5>ke7 0924ff77-ef51-435b-b90d-
>> 50bfbf2e8de7 /rhev/data-center/mnt/glusterSD/vbgsan02:_fs02/0924ff77-
>> ef51-435b-b90d-50bfbf2e8de7/dom_md/ids
>> 2017-07-30 10:49:31+0200 1766275 [10496]: leader1 delta_acquire_begin
>> error -226 lockspace 0924ff77-ef51-435b-b90d-50bfbf2e8de7 host_id 1
>> 2017-07-30 10:49:31+0200 1766275 [10496]: leader2 path /rhev/data-
>> center/mnt/glusterSD/vbgsan02:_fs02/0924ff77-ef51-435b-b90d-
>> 50bfbf2e8de7/dom_md/ids offset 0
>> 2017-07-30 10:49:31+0200 1766275 [10496]: leader3 m 12212010 v 30003 ss
>> 512 nh 0 mh 4076 oi 1 og 2031079063 lv 0
>> 2017-07-30 10:49:31+0200 1766275 [10496]: leader4 sn 0924ff77-ef51-
>> 435b-b90d-50bfbf2e<D5>ke7 rn <93><F6>7^\afa5-3a91-415b-a04c-
>> 221d3e060163.vbgkvm01.a ts 4351980 cs eefa4dd7
>> 2017-07-30 10:49:32+0200 1766276 [1171]: s310751 add_lockspace fail
>> result -226
>>
>>
>> vdsm logs doesnt have any errors and engine.log does not have any
>> errors.
>>
>> And if i check the ids file manually. I can see that everything in it
>> is correct except for the first host in the cluster where the space
>> name and host id is broken.
>>
>>
>> /Johan
>>
>> On Sun, 2017-07-30 at 11:18 +0300, Maor Lipchuk wrote:
>>> Hi Johan,
>>>
>>> Can you please share the vdsm and engine logs.
>>>
>>> Also, it won't harm to also get the sanlock logs just in case sanlock
>>> was configured to save all debugging in a log file (see
>>> http://people.redhat.com/teigland/sanlock-messages.txt)).
>>> Try to share the sanlock ouput by running  'sanlock client status',
>>> 'sanlock client log_dump'.
>>>
>>> Regards,
>>> Maor
>>>
>>> On Thu, Jul 27, 2017 at 6:18 PM, Johan Bernhardsson <johan at kafit.se>
>>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > The ids file for sanlock is broken on one setup. The first host id
>>> > in
>>> > the file is wrong.
>>> >
>>> > From the logfile i have:
>>> >
>>> > verify_leader 1 wrong space name 0924ff77-ef51-435b-b90d-
>>> > 50bfbf2e�ke7
>>> > 0924ff77-ef51-435b-b90d-50bfbf2e8de7 /rhev/data-
>>> > center/mnt/glusterSD/
>>> >
>>> >
>>> >
>>> > Note the broken char in the space name.
>>> >
>>> > This also apears. And it seams as the hostid too is broken in the
>>> > ids
>>> > file:
>>> >
>>> > leader4 sn 0924ff77-ef51-435b-b90d-50bfbf2e�ke7 rn ��7 afa5-3a91-
>>> > 415b-
>>> > a04c-221d3e060163.vbgkvm01.a ts 4351980 cs eefa4dd7
>>> >
>>> > Note the broken chars there as well.
>>> >
>>> > If i check the ids file with less or strings the first row where my
>>> > vbgkvm01 host are. That has broken chars.
>>> >
>>> > Can this be repaired in some way without taking down all the
>>> > virtual
>>> > machines on that storage?
>>> >
>>> >
>>> > /Johan
>>> > _______________________________________________
>>> > Users mailing list
>>> > Users at ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users