Roy Golan has submitted this change and it was merged. Change subject: core: throttle running of VMs (#843058) ...................................................................... core: throttle running of VMs (#843058) https://bugzilla.redhat.com/show_bug.cgi?id=843058 Bulk running of VMs regulary falls short in running all VMs. The reason is the VdsSelector counts and reserves memory using: a) host commited memory: # of VMs * each VM static mem b) pending VM count memory : # of VMs to run * their memory count ; a shared state variable pending VM count is increased by the end of RunVm and decreased by VdsUpdateRunTimeInfo i.e two independent processes, one as a command thread and the other is a scheduled quartz job thread. As long as the VdsUpdateRunTimeInfo doesn't run, theres an overflow in the calculation of the free memory to run VM which causes a host to be falsely not selected to run the VM. The throttler here is using the shared-state between the VdsUpdateRunTimeInfo and the RunVmCommands, the pending VM count and is using a wait/notify semantics to let the VdsUpdateRunTimeInfo interleave, if we don't have enough memory run the VM. - Tests - setup: 1 Host with 128Gb mem with 120 VMs and cluster 200% over commit 1 Host with 8Gb mem, 3.0 pool and cluster, 10 VMs 2 Host with 4Gb mem, 3.1 pool and cluster, 5 VMs on the 128Gb host I activated 100 VMs with 3G and 82 of them got online. without the fix I hit 51 limit. with no over commit I was able to run also 67 and without the fix also 50 multiple run 10 VMs - prior to the fix the 9th and 10th can't locate a host to run on. after the fix all 10 VMs complete the run. In the 9th VM run the selector waits for the update to decrease memory and continue after less than a second. migrate VMs back and forth of 5 VMs with no issues. VmPool - same scheduling effects. Change-Id: I076ede6cba919bc61f7546d7b29ef436eb6d3375 Signed-off-by: Roy Golan <rgolan@redhat.com> --- M backend/manager/dbscripts/upgrade/pre_upgrade/0000_config.sql M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/MigrateVmCommand.java A backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/NonWaitingDelayer.java M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/RunVmCommand.java M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/RunVmCommandBase.java A backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/RunVmDelayer.java M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VdsEventListener.java A backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VdsFreeMemoryChecker.java M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VdsSelector.java M backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/VmPoolCommandBase.java M backend/manager/modules/bll/src/test/java/org/ovirt/engine/core/bll/RunVmCommandTest.java M backend/manager/modules/common/src/main/java/org/ovirt/engine/core/common/businessentities/IVdsAsyncCommand.java M backend/manager/modules/common/src/main/java/org/ovirt/engine/core/common/config/ConfigValues.java M backend/manager/modules/vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/VdsManager.java A backend/manager/modules/vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/VdsMonitor.java 15 files changed, 242 insertions(+), 33 deletions(-) Approvals: Roy Golan: Verified; Looks good to me, approved -- To view, visit http://gerrit.ovirt.org/7204 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: merged Gerrit-Change-Id: I076ede6cba919bc61f7546d7b29ef436eb6d3375 Gerrit-PatchSet: 10 Gerrit-Project: ovirt-engine Gerrit-Branch: master Gerrit-Owner: Roy Golan <rgolan@redhat.com> Gerrit-Reviewer: Allon Mureinik <amureini@redhat.com> Gerrit-Reviewer: Barak Azulay <bazulay@redhat.com> Gerrit-Reviewer: Doron Fediuck <dfediuck@redhat.com> Gerrit-Reviewer: Michael Kublin <mkublin@redhat.com> Gerrit-Reviewer: Moti Asayag <masayag@redhat.com> Gerrit-Reviewer: Omer Frenkel <ofrenkel@redhat.com> Gerrit-Reviewer: Roy Golan <rgolan@redhat.com> Gerrit-Reviewer: Yair Zaslavsky <yzaslavs@redhat.com> Gerrit-Reviewer: oVirt Jenkins CI Server