Re: oVirt and NetApp NFS storage
I know 2 approaches. 1. Use NFS hard mounting option - it will never give error to sanlock and it will be waiting until NFS is recovered (never tries this one, but in theory might work) 2. Change the default sanlock timeout (last time I tried that - it didn't work) . You might need help from Sandro or Sahina for that option. Best Regards, Strahil NikolovOn Apr 18, 2019 11:45, klaasdemter@gmail.com wrote:
Hi,
I got a question regarding oVirt and the support of NetApp NFS storage. We have a MetroCluster for our virtual machine disks but a HA-Failover of that (active IP gets assigned to another node) seems to produce outages too long for sanlock to handle - that affects all VMs that have storage leases. NetApp says a "worst case" takeover time is 120 seconds. That would mean sanlock has already killed all VMs. Is anyone familiar with how we could setup oVirt to allow such storage outages? Do I need to use another type of storage for my oVirt VMs because that NFS implementation is unsuitable for oVirt?
Greetings
Klaas _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TSJJKK5UG57CCF...
Hi, NFS hard mounts should not help with sanlock. It should just kill the VMs after 60 seconds same as a soft mount. The sanlock timeout: I asked RedHat support about that in a case for my RHV environment. They said this should not be changed. Only thing that came to my mind: disable storage leases in case of planned storage maintenance. Greetings Klaas On Thu, Apr 18, 2019, 19:10 Strahil <hunter86_bg@yahoo.com> wrote:
I know 2 approaches. 1. Use NFS hard mounting option - it will never give error to sanlock and it will be waiting until NFS is recovered (never tries this one, but in theory might work) 2. Change the default sanlock timeout (last time I tried that - it didn't work) . You might need help from Sandro or Sahina for that option.
Best Regards, Strahil NikolovOn Apr 18, 2019 11:45, klaasdemter@gmail.com wrote:
Hi,
I got a question regarding oVirt and the support of NetApp NFS storage. We have a MetroCluster for our virtual machine disks but a HA-Failover of that (active IP gets assigned to another node) seems to produce outages too long for sanlock to handle - that affects all VMs that have storage leases. NetApp says a "worst case" takeover time is 120 seconds. That would mean sanlock has already killed all VMs. Is anyone familiar with how we could setup oVirt to allow such storage outages? Do I need to use another type of storage for my oVirt VMs because that NFS implementation is unsuitable for oVirt?
Greetings
Klaas _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TSJJKK5UG57CCF...
So after a little wild discussion with support it boiled down to this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1705289 which suggests it may be a good idea to have sanlock io_timeout configureable :)
I too am suffering from this. Whilst a proper oVirt-configuration item would be ideal, the RFE suggests that there is a manual work-around for setting sanlock's io_timeout, albeit not recommended. Could anyone share what it is? It seems clear from the sanlock code that io_timeout is the only configurable item, and that the renewal fail timeout is 8 x io_timeout. In that case, increasing the io_timeout from the default of 10 to 15 should cover me. https://pagure.io/sanlock/blob/master/f/src/timeouts.h https://pagure.io/sanlock/blob/master/f/src/timeouts.c Searching suggests two different ways. Firstly, putting, e.g. SANLOCKOPTS="-o 15", in /etc/sysconfig/sanlock, though the sanlock docs do not confirm that should work and the search hits are all very old. Secondly, as oVirt is using libvirt, setting io_timeout=15 in /etc/libvirt/qemu-sanlock.conf. Does anyone know if either of those work? Or if there's another way? Thanks, Jon
my current idea of a workaround is to disable storage leases before netapp upgrades, and re-enable them after upgrades are done. python sdk should make this fairly easy (https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/set_vm_le... and https://gerrit.ovirt.org/c/99712/)
So for all lease-holding vms, push the lease to a different storage domain (on different storage hardware), apply the upgrade and then push the leases back? And that can be done whilst the VMs are running? Leases must be frequently renewed, so I guess no particular reason why not. Does that work for the SPM and hosted engine as well? I'm guessing this doesn't help with an unmanaged contoller failover. Anecdotally, that seems to happen a bit faster for me than a managed one, which is also odd.
So for all lease-holding vms, push the lease to a different storage domain (on different storage hardware), apply the upgrade and then push the leases back? And that can be done whilst the VMs are running? Leases must be frequently renewed, so I guess no particular reason why not.
If you have the storage leases on another hardware then you do not have the same problem I have; unless you are talking about upgrading the hardware of the storage domain that holds the leases. General: you can change the lease on a running VM. But (speaking for 4.2) you have to set it to no lease first.
Does that work for the SPM and hosted engine as well?
I think the hosted engine has it's own kind of storage lease, but I can't say anything to that. SPM does not have a storage lease it should be hardware (ie it's one of the hypervisors) not a VM.
I'm guessing this doesn't help with an unmanaged contoller failover. Anecdotally, that seems to happen a bit faster for me than a managed one, which is also odd.
Yeah, all my ideas are just for planned maintenance.
participants (4)
-
jon＠well.ox.ac.uk -
Klaas Demter -
klaasdemter＠gmail.com
-
Strahil