[ovirt-users] Re: Random reboots

On Thu, Feb 17, 2022 at 11:20 AM Pablo Olivera <p.olivera(a)telfy.com&gt; wrote: You can do one of these: 1. Use lower timeouts on the NFS server mount, so the NFS server at the same time the sanlock lease times out. 2. Use larger sanlock timeout so sanlock lease time out when the NFS server times out. 3. Both 1 and 2 The problem is that NFS timeouts are not predictable. In the past we used: "timeo=600,retrans=6" which can lead to 21 minutes timeout, but practically we saw up to a 30 minutes timeout. In https://github.com/oVirt/vdsm/commit/672a98bbf3e55d1077669f06c37305185fbd... we change this to the recommended seting: "timeo=100,retrans=3" Which according to the docs, should fail in 60 seconds if all retries fail. But practically we saw up to 270 seconds timeout with this setting, which does not play well with sanlock. We assumed that the timeout value should not be less than sanlock io timeout (10 seconds) but I'm not sure this assumption is correct. You can smaller timeout value in engine storage domain "custom connections parameters" - Retransmissions - mapped to "retrans" mount option - Timeout (deciseconds) - mapped to "timeo" mount option For example: Retransmissions: 3 Timeout: 5 Theoretically this will behave like this: 00:00 retry 1 (5 seconds timeout) 00:10 retry 2 (10 seconds timeout) 00:30 retry 3 (15 seconds timeout) 00:45 request fail But based on what we see with the defaults, this is likely to take more time. If it fails before 140 seconds, the VM will be killed and the host will not reboot. The other way is to increase sanlock timeout, in vdsm configuration. note that changing sanlock timeout requires also changing other settings (e.g. spm:watchdog_interval). Add this file on all hosts: $ cat /etc/vdsm/vdsm.conf.d/99-local.conf [spm] # If enabled, montior the SPM lease status and panic if the lease # status is not expected. The SPM host will lose the SPM role, and # engine will select a new SPM host. (default true) # watchdog_enable = true # Watchdog check internal in seconds. The recommended value is # sanlock:io_timeout * 2. (default 20) watchdog_interval = 40 [sanlock] # I/O timeout in seconds. All sanlock timeouts are computed based on # this value. Using larger timeout will make VMs more resilient to # short storage outage, but increase VM failover time and the time to # acquire a host id. For more info on sanlock timeouts please check # sanlock source: # https://pagure.io/sanlock/raw/master/f/src/timeouts.h. If your # storage requires larger timeouts, you can increase the value to 15 # or 20 seconds. If you change this you need to update also multipath # no_path_retry. For more info onconfiguring multipath please check # /etc/multipath.conf.oVirt is tested only with the default value (10 # seconds) io_timeout = 20 You can check https://github.com/oVirt/vdsm/blob/master/doc/io-timeouts.md to learn more about sanlock timeouts. Alternatively, you can make a small change in NFS timeout and small change in sanlock timeout to make them work better together. All this is of course to handle the case when the NFS server is not accessible, but this is something that should not happen in a healthy cluster. You need to check why the server was not accessible and fix this problem. Nir