[ovirt-users] HA VM Lease failure with full data storage domain
OK, so the data storage domain on a cluster filled up to the point that
the OS refused to allocate any more space.
This happened because I tried to create a new prealloc'd disk from the
Admin WebUI. The disk creation claims to be completed successfully,
I've not tried to use that disk yet, but due to a timeout with the
storage domain in question the engine began trying to fence all of the
HA VMs.
The fencing failed for all of the HA VMs leaving them in a powered off
state. Despite all of the HA VMs being up at the time, so no
reallocation of the leases should have been necessary. Attempting to
restart them manually from the Admin WebUI failed. With the original
host they were running on complaining about "no space left on device",
and the other hosts claiming that the original host still held the VM
lease.
After cleaning up some old snapshots, the HA VMs would still not boot.
Toggling the High Availability setting for each one and allowing the
lease to be removed from the storage domain was required to get the VMs
to start again. Re-enabling the High Availability setting there after
fixed the lease issue. But now some, not all, of the HA VMs are still
throwing "no space left on device" errors when attempting to start
them. The others are working just fine even with their HA lease
enabled.
My questions are:
1. Why does oVirt claim to have a constantly allocated HA VM lease on
the storage domain when it's clearly only done while the VM is running?
2. Why does oVirt deallocate the HA VM lease when performing a fencing
operation?
3. Why can't oVirt clear the old HA VM lease when the VM is down and
the storage pool has space available? (How much space is even needed?
The leases section of the storage domain in the Admin WebUI doesn't
contain any useful info beyond the fact that a lease should exist for a
VM even when it's off.)
4. Is there a better way to force start a HA VM when the lease is old
and the VM is powered off?
5. Should I file a bug on the whole HA VM failing to reacquire a lease
on a full storage pool?
-Patrick Hibbs