Hi, I'd say your scenario is exactly what the storage leases are for. I can tell you that it used to work in 4.3, but I haven't tested/needed the feature in quite a while :) Maybe open a bugzilla and attach relevant logs to get the developer attention. Greetings Klaas On 8/5/21 4:42 PM, Gianluca Cecchi wrote:

Hi, I always thought the SPM role is also "managed" by a storage lease :) But that does not seem to be the case. So this means a storage lease is only useful if the host is not the SPM? If the SPM host is completely unreachable, not via OS, not via power management, then the storage lease won't help to restart VMs on other hosts automatically? This is definitely something I did not consider when building my environment. Greetings Klaas On 8/9/21 6:25 PM, Nir Soffer wrote: > On Thu, Aug 5, 2021 at 5:45 PM Gianluca Cecchi > <gianluca.cecchi(a)gmail.com&gt; wrote: >> Hello, >> supposing latest 4.4.7 environment installed with an external engine and two hosts, one in one site and one in another site. >> For storage I have one FC storage domain. >> I try to simulate a sort of "site failure scenario" to see what kind of HA I should expect. >> >> The 2 hosts have power mgmt configured through fence_ipmilan. >> >> I have 2 VMs, one configured as HA with lease on storage (Resume Behavior: kill) and one not marked as HA. >> >> Initially host1 is SPM and it is the host that runs the two VMs. >> >> Fencing of host1 from host2 initially works ok. I can test also from command line: >> # fence_ipmilan -a 10.10.193.152 -P -l my_fence_user -A password -L operator -S /usr/local/bin/pwd.sh -o status >> Status: ON >> >> On host2 I then prevent reaching host1 iDRAC: >> firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -d 10.10.193.152 -p udp --dport 623 -j DROP >> firewall-cmd --direct --add-rule ipv4 filter OUTPUT 1 -j ACCEPT > Why do you need to prevent access from host1 to host2? Hosts do not > access each other unless you migrate vms between hosts. > >> so that: >> >> # fence_ipmilan -a 10.10.193.152 -P -l my_fence_user -A password -L operator -S /usr/local/bin/pwd.sh -o status >> 2021-08-05 15:06:07,254 ERROR: Failed: Unable to obtain correct plug status or plug is not available >> >> On host1 I generate panic: >> # date ; echo 1 > /proc/sys/kernel/sysrq ; echo c > /proc/sysrq-trigger >> Thu Aug 5 15:06:24 CEST 2021 >> >> host1 correctly completes its crash dump (kdump integration is enabled) and reboots, but I stop it at grub prompt so that host1 is unreachable from host2 point of view and also power fencing not determined > Crashing the host and preventing it from booting is fine, but isn't it > simpler to stop the host using power management? > >> At this point I thought that VM lease functionality would have come in place and host2 would be able to re-start the HA VM, as it is able to see that the lease is not taken from the other host and so it can acquire the lock itself.... > Once host1 disappears from the system, engine should detect that the HA VM > is at unknown status, and start it on the other host. > > But you kill the SPM, and without SPM some operation cannot > work until a new SPM is selected. And for the SPM we don't have a way > to start it on another host *before* the old SPM host reboot, and we can > verify that the old host is not the SPM. > >> Instead it goes through the attempt to power fence loop >> I wait about 25 minutes without any effect but continuous attempts. >> >> After 2 minutes host2 correctly becomes SPM and VMs are marked as unknown > I wonder how host2 became the SPM. This should not be possible before > host 1 is rebooted. Did you use "Confirm host was rebooted" in engine? > >> At a certain point after the failures in power fencing host1, I see the event: >> >> Failed to power fence host host1. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >> >> If I select host and choose "Confirm Host Has Been Rebooted", then the two VMs are marked as down and the HA one is correctly booted by host2. >> >> But this requires my manual intervention. > So you host2 became the SPM after you chose: "Confirm Host Has Been Rebooted"? > >> Is the behavior above the expected one or the use of VM leases should have allowed host2 to bypass fencing inability and start the HA VM with lease? Otherwise I don't understand the reason to have the lease itself at all.... > The vm lease allows engine to start HA VM on another host when it cannot > access the original host the VM was running on. > > The VM can start only if it is not running on the original host. If > the VM is running > it will keep the lease live, and other hosts will not be able to acquire it. > > I suggest you file an ovirt-engine bug with clear instructions on how > to reproduce > the issue. > > You can check this presentation on this topic: > https://www.youtube.com/watch?v=WLnU_YsHWtU > > Nir > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/J3QWSUZTYHZ...

Okay, so the SPM is only blocking me if I have thin provisioned block storage that needs extending. This is luckily not the case because we primarily use NFS. It is something though I have never thought about, in my head the storage leases completely solved the "host crashes and power management does not answer" until I read your mail :) Thanks for the detailed explanation Nir! Greetings Klaas On 8/10/21 11:20 AM, Nir Soffer wrote: > On Tue, Aug 10, 2021 at 12:05 PM Klaas Demter <klaasdemter(a)gmail.com&gt; wrote: >> I always thought the SPM role is also "managed" by a storage lease :) > The SPM is using a storage lease to ensure we have only one SPM. But due to > the master mount, we cannot start new SPM even if the old SPM does not hold > the lease, since it will corrupt the master filesystem, used to keep SPM tasks. > >> But that does not seem to be the case. >> >> So this means a storage lease is only useful if the host is not the SPM? >> If the SPM host is completely unreachable, not via OS, not via power >> management, then the storage lease won't help to restart VMs on other >> hosts automatically? This is definitely something I did not consider >> when building my environment. > Starting VMs should not depend on the SPM, this is the basic design. If issues > with SPM beaks starting VMs, this is a bug that we need to fix. > > The only known dependency is extending thin provisioning disks on block storage. > Without the SPM this cannot happen since the SPM is the only host that > can extend > the logical volumes. > >> On 8/9/21 6:25 PM, Nir Soffer wrote: >>> On Thu, Aug 5, 2021 at 5:45 PM Gianluca Cecchi >>> <gianluca.cecchi(a)gmail.com&gt; wrote: >>>> Hello, >>>> supposing latest 4.4.7 environment installed with an external engine and two hosts, one in one site and one in another site. >>>> For storage I have one FC storage domain. >>>> I try to simulate a sort of "site failure scenario" to see what kind of HA I should expect. >>>> >>>> The 2 hosts have power mgmt configured through fence_ipmilan. >>>> >>>> I have 2 VMs, one configured as HA with lease on storage (Resume Behavior: kill) and one not marked as HA. >>>> >>>> Initially host1 is SPM and it is the host that runs the two VMs. >>>> >>>> Fencing of host1 from host2 initially works ok. I can test also from command line: >>>> # fence_ipmilan -a 10.10.193.152 -P -l my_fence_user -A password -L operator -S /usr/local/bin/pwd.sh -o status >>>> Status: ON >>>> >>>> On host2 I then prevent reaching host1 iDRAC: >>>> firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -d 10.10.193.152 -p udp --dport 623 -j DROP >>>> firewall-cmd --direct --add-rule ipv4 filter OUTPUT 1 -j ACCEPT >>> Why do you need to prevent access from host1 to host2? Hosts do not >>> access each other unless you migrate vms between hosts. >>> >>>> so that: >>>> >>>> # fence_ipmilan -a 10.10.193.152 -P -l my_fence_user -A password -L operator -S /usr/local/bin/pwd.sh -o status >>>> 2021-08-05 15:06:07,254 ERROR: Failed: Unable to obtain correct plug status or plug is not available >>>> >>>> On host1 I generate panic: >>>> # date ; echo 1 > /proc/sys/kernel/sysrq ; echo c > /proc/sysrq-trigger >>>> Thu Aug 5 15:06:24 CEST 2021 >>>> >>>> host1 correctly completes its crash dump (kdump integration is enabled) and reboots, but I stop it at grub prompt so that host1 is unreachable from host2 point of view and also power fencing not determined >>> Crashing the host and preventing it from booting is fine, but isn't it >>> simpler to stop the host using power management? >>> >>>> At this point I thought that VM lease functionality would have come in place and host2 would be able to re-start the HA VM, as it is able to see that the lease is not taken from the other host and so it can acquire the lock itself.... >>> Once host1 disappears from the system, engine should detect that the HA VM >>> is at unknown status, and start it on the other host. >>> >>> But you kill the SPM, and without SPM some operation cannot >>> work until a new SPM is selected. And for the SPM we don't have a way >>> to start it on another host *before* the old SPM host reboot, and we can >>> verify that the old host is not the SPM. >>> >>>> Instead it goes through the attempt to power fence loop >>>> I wait about 25 minutes without any effect but continuous attempts. >>>> >>>> After 2 minutes host2 correctly becomes SPM and VMs are marked as unknown >>> I wonder how host2 became the SPM. This should not be possible before >>> host 1 is rebooted. Did you use "Confirm host was rebooted" in engine? >>> >>>> At a certain point after the failures in power fencing host1, I see the event: >>>> >>>> Failed to power fence host host1. Please check the host status and it's power management settings, and then manually reboot it and click "Confirm Host Has Been Rebooted" >>>> >>>> If I select host and choose "Confirm Host Has Been Rebooted", then the two VMs are marked as down and the HA one is correctly booted by host2. >>>> >>>> But this requires my manual intervention. >>> So you host2 became the SPM after you chose: "Confirm Host Has Been Rebooted"? >>> >>>> Is the behavior above the expected one or the use of VM leases should have allowed host2 to bypass fencing inability and start the HA VM with lease? Otherwise I don't understand the reason to have the lease itself at all.... >>> The vm lease allows engine to start HA VM on another host when it cannot >>> access the original host the VM was running on. >>> >>> The VM can start only if it is not running on the original host. If >>> the VM is running >>> it will keep the lease live, and other hosts will not be able to acquire it. >>> >>> I suggest you file an ovirt-engine bug with clear instructions on how >>> to reproduce >>> the issue. >>> >>> You can check this presentation on this topic: >>> https://www.youtube.com/watch?v=WLnU_YsHWtU >>> >>> Nir >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/J3QWSUZTYHZ... >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7RYGVJO52IH...