3 p.m.
Greetings,
we have recently installed ovirt as a hosted-engine with high availability on six nodes
over NFS storage (no Gluster), with power management through an on-board IPMI device, and
the setup was successful. All the nodes (from Supermicro) are identical in every aspect,
so no hardware differences exist and no modifications to the servers' hardware were
performed. The hosted-engine was deployed on a second host, where two of the six hosts
only were required to host the HE VM.
The network interface on each node is bonded between two physical fiber optics NICs in
LACP mode with a VLAN on top, serving as the sole networking interface for the
server/node, no separate VM or storage networks were needed, as the host OS, hosted-engine
vm, and storage are required to be on the same network and VLAN.
We started by testing the high-availability of the hosted-engine VM (as it was deployed on
two of the six nodes) by rebooting or powering off one of the hosts, and the VM would
migrate successfully to the second HE node. The main goal of our experiments is to test
the robustness of the setup, as it is required for the cluster to remain functional even
when up to two hosts are brought down (whether due to a network or power issue), however,
when rebooting or powering off one of the hosts, the HE VM goes down and takes the entire
cluster with it, where we can't even access the web portal. Once the host is rebooted,
the HE VM and the cluster becomes functional again. Sometimes the HE VM stays down for a
set amount of time (5 to 6 minutes) and then goes back up, and sometimes it goes down
until the problematic host is back up. This behavior happens to other VMs as well not the
the HE.
We suspected an issue with the NFS storage, however, during ovirt operation it is being
mounted properly over /rhev/data-center/mnt/<nfs:directory>, while the expected
behavior is for the cluster to stay operational and any other VMs to be migrated to other
hosts. During one of the tests, we tried to mount the NFS storage on a different directory
and there was no problem, we were even able to perform commands such as ls without any
issues, as well as writing a text file at the directory's root, and be able to modify
it normally.
We suspected a couple of things the first being that the HE is unable to fence the
problematic host (the one we took down), however, power management is setup properly.
The other thing we suspected is the cluster hosts (after taking down one of them) are
unable to acquire storage lease, which is weird since the host in question is down and
non-operational, hence no locks should be in place. The reason behind this suspicion is
the following two errors that we receive frequently when one host or more goes down from
the engine\ovirt-engine\engine.log file:
1- "EVENT_ID: VM_DOWN_ERROR(119), VM HostedEngine is down with error. Exit message:
resource busy: Failed to acquire lock: Lease is held by another host."
2- "[<id>] Command 'GetVmLeaseInfoVDSCommand(
VmLeaseVDSParameters:{expectedEngineErrors='[NoSuchVmLeaseOnDomain]',
storagePoolId='<pool-id>', ignoreFailoverLimit='false',
leaseId='<lease-id>', storageDomainId='<domain-id>'})'
execution failed: IRSGenericException: IRSErrorException: No such lease:
'lease=<lease-id>'"
This is a third warning from the /var/log/vdsm/vdsm.log file
1- "WARN (check/loop) [storage.check] Checker
'/rhev/data-center/mnt/<nfs-domain:/directory>/<id>/dom_md/metadata'
is blocked for 310.00 seconds (check:265)"
All the tests are done without setting nodes into maintenance mode as we are simulating an
emergency situation. No HE configuration were modified via the config-engine command, the
default values are used.
Is this a normal behavior? Are we missing something? Do we need to tweak a certain
configuration using the config-engine command to get a better behavior (e.g., shorter down
period)?
Best regards