[Users] sanlock leases on VM disks

[José Luis Sanz Boixader]

On 01/17/2014 11:43 PM, Itamar Heim wrote:
> On 01/10/2014 08:44 PM, José Luis Sanz Boixader wrote:
>> I have an oVirt testing setup with 3 hosts running for a few weeks:
>> CentOS 6.4, oVirt 3.3.1, VDSM 4.13.0, iSCSI based storage domain.
>>
>> I have just realized that sanlock has no leases on VM disks, so nothing
>> prevents vdsm/libvirt from starting a VM on two different hosts,
>> corrupting disk data. I know that something has to go wrong on oVirt
>> engine to do it, but I've manually forced some errors ("Setting Host
>> state to Non-Operational", "VM xxxx is not responding") for a "Highly
>> available" VM and oVirt engine started that VM on another host. oVirt
>> engine was not aware, but the VM was running on two hosts.
>>
>> I think this is a job for libvirt/sanlock/wdmd, but libvirt is not
>> receiving "lease" tags for disks when creating domains. I think it
>> should.
>> What's left in my config? What am I doing wrong?
>>
>> Thanks
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
> we started introducing sanlock carefully, first to SPM nodes. in 3.4
> to hosted ovirt-engine node, and looking to add it to VMs/disks as
> well going forward.
>
> I don't remember if we have a config option to enable this, but  you
> can make this work via a custom hook at this point at vm/disk level,
> and we would love feedback on this.
>
> Thanks,
>    Itamar
>

Looking into vdsm code, I've found that there's already code for sanlock
on VM disks, but it has been disabled by default
[https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=838802].
I guess it was disabled because you can't hot attach/dettach disks to VM
while running. But I prefer to enable it, as data protection is critical
in a SAN environment.

To enable it, you need to include this in /etc/vdsm/vdsm.conf at every
host in your setup.
...
[irs]
use_volume_leases = true
...
and after that, you'll need to restart vdsmd. Ensure that
/etc/libvirt/qemu.conf says lock_manager="sanlock".
VMs that were already running will not be modified, and thus not
protected. Restart them to get sanlock leases on those disks.

To confirm that it is properly running, connect to a host and type:
# sanlock client status
and you'll get some output like this, listing VMs running on that host
and its disk leases:

daemon dc7e06a0-18bb-4f68-9ea6-883dda883ef2.server
p -1 helper
p -1 listener
p 10629 Vicent
p -1 status
s
e9a91ad7-2bd3-4c98-a171-88324bc87a09:2:/dev/e9a91ad7-2bd3-4c98-a171-88324bc87a09/ids:0
r
e9a91ad7-2bd3-4c98-a171-88324bc87a09:1c7005ad-3d33-4c8f-9c99-2eef7be865f3:/dev/e9a91ad7-2bd3-4c98-a171-88324bc87a09/leases:123731968:6
p 10629
r
e9a91ad7-2bd3-4c98-a171-88324bc87a09:978b3630-f491-45a3-9826-e9ab6a744e72:/dev/e9a91ad7-2bd3-4c98-a171-88324bc87a09/leases:127926272:6
p 10629

Which tells that domain "Vicent" with proc id 10629 has two disks with
leases.
No other host will be able to run that VM, even in a engine error event,
because it could not get leases to use those disks.

Migration of VM also works, and the destination host gets and acquires
the disk leases.
All this has been tested with oVirt 3.3.1 release.