Change in ovirt-engine[master]: core: throttle running of VMs (#843058)

[rgolan at redhat.com]

Roy Golan has submitted this change and it was merged.

Change subject: core: throttle running of VMs (#843058)
......................................................................


core: throttle running of VMs (#843058)

https://bugzilla.redhat.com/show_bug.cgi?id=843058

Bulk running of VMs regulary falls short in running all VMs. The reason is
the VdsSelector counts and reserves memory using:
a) host commited memory:     # of VMs * each VM static mem
b) pending VM count memory : # of VMs to run * their memory count ; a shared state variable

pending VM count is increased by the end of RunVm and decreased by VdsUpdateRunTimeInfo
i.e two independent processes, one as a command thread and the other is a scheduled quartz job thread.

As long as the VdsUpdateRunTimeInfo doesn't run, theres an overflow in the calculation
of the free memory to run VM which causes a host to be falsely not selected to run the VM.

The throttler here is using the shared-state between the VdsUpdateRunTimeInfo and the RunVmCommands, the
pending VM count and is using a wait/notify semantics to let the VdsUpdateRunTimeInfo interleave,
if we don't have enough memory run the VM.

- Tests -

setup: 1 Host with 128Gb mem with 120 VMs and cluster 200% over commit
       1 Host with 8Gb mem, 3.0 pool and cluster, 10 VMs
       2 Host with 4Gb mem, 3.1 pool and cluster, 5 VMs

on the 128Gb host I activated 100 VMs with 3G and 82 of them got online. without the fix I hit 51 limit.
with no over commit I was able to run also 67 and without the fix also 50

multiple run 10 VMs - prior to the fix the 9th and 10th can't locate a host to run on.
 after the fix all 10 VMs complete the run. In the 9th VM run the selector waits for the update to decrease memory
 and continue after less than a second.

migrate VMs back and forth of 5 VMs with no issues.

VmPool - same scheduling effects.

Change-Id: I076ede6cba919bc61f7546d7b29ef436eb6d3375
Signed-off-by: Roy Golan <rgolan at redhat.com>
---
M backend/manager/dbscripts/upgrade/pre_upgrade/0000_config.sql
M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/MigrateVmCommand.java
A backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/NonWaitingDelayer.java
M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/RunVmCommand.java
M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/RunVmCommandBase.java
A backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/RunVmDelayer.java
M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VdsEventListener.java
A backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VdsFreeMemoryChecker.java
M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VdsSelector.java
M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VmPoolCommandBase.java
M backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/RunVmCommandTest.java
M backend/manager/modules/common/src/main/java/org/ovirt/engine/core/common/businessentities/IVdsAsyncCommand.java
M backend/manager/modules/common/src/main/java/org/ovirt/engine/core/common/config/ConfigValues.java
M backend/manager/modules/vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/VdsManager.java
A backend/manager/modules/vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/VdsMonitor.java
15 files changed, 242 insertions(+), 33 deletions(-)

Approvals:
  Roy Golan: Verified; Looks good to me, approved


--
To view, visit http://gerrit.ovirt.org/7204
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I076ede6cba919bc61f7546d7b29ef436eb6d3375
Gerrit-PatchSet: 10
Gerrit-Project: ovirt-engine
Gerrit-Branch: master
Gerrit-Owner: Roy Golan <rgolan at redhat.com>
Gerrit-Reviewer: Allon Mureinik <amureini at redhat.com>
Gerrit-Reviewer: Barak Azulay <bazulay at redhat.com>
Gerrit-Reviewer: Doron Fediuck <dfediuck at redhat.com>
Gerrit-Reviewer: Michael Kublin <mkublin at redhat.com>
Gerrit-Reviewer: Moti Asayag <masayag at redhat.com>
Gerrit-Reviewer: Omer Frenkel <ofrenkel at redhat.com>
Gerrit-Reviewer: Roy Golan <rgolan at redhat.com>
Gerrit-Reviewer: Yair Zaslavsky <yzaslavs at redhat.com>
Gerrit-Reviewer: oVirt Jenkins CI Server