MoM - Not working ? How to use it ?
Hello, I'm trying to test Memory balloon useful functionality, but l'm unable to make it work successfuly. Firstly, I enable the MOM on my Cluster: Memory Balloon Enable Memory Balloon Optimization Secondly, I configure a VM virtual machine like this: On system tab: Memory Size : 28672 MB Maximum memory : 28672 MB On Ressource allocation tab: Memory Allocation: Physical Memory Guaranteed : 2048 MB Memory Balloon Device: Enabled Thirdly, I test 1- I boot the VM. After boot VM consume 200MB 2- I lauch perl program which consume 26GB. 3- During this, Host RAM usage is near 90% 4- I stop the perl program, and on guest side, RAM usage drop down to 200Mo 5- On host side, RAM usage keep to 24 - 26GB. I wait some time to show if the qemu-vm process drop is host-ram-usage. Ram usage keep to 24GB. 6- I try to launch another VM like the first one (but with 8GB RAM and 2GB Guaranteed) , on the host (to "force" Mom ?) but I'm unable to lauch it, Host free mem is insufficient I change MoM log to debug to show something: 2018-07-31 11:49:31,887 - mom.RPCServer - INFO - ping() 2018-07-31 11:49:31,888 - mom.RPCServer - INFO - getStatistics() 2018-07-31 11:49:31,888 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. 2018-07-31 11:49:43,313 - mom.VdsmRpcBase - DEBUG - VM List: [u'9632443f-d302-43f7-a279-778e64ee98f4'] 2018-07-31 11:49:43,465 - mom.VdsmRpcBase - DEBUG - Memory stats: {'swap_out': 0, 'swap_usage': 0, 'mem_free': 28221020, 'major_fault': 0, 'swap_in': 0, 'swap_total': 0, 'mem_available': 28650472, 'minor_fault': 99, 'mem_unused': 28221020} 2018-07-31 11:49:43,468 - mom.Monitor - DEBUG - Collector <mom.Collectors.GuestIoTuneOptional.GuestIoTuneOptional instance at 0x167de18> did not return any data 2018-07-31 11:49:43,931 - mom.RPCServer - INFO - ping() 2018-07-31 11:49:43,932 - mom.RPCServer - INFO - getStatistics() 2018-07-31 11:49:43,932 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. 2018-07-31 11:49:44,258 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. 2018-07-31 11:49:44,284 - mom.Evaluator - DEBUG - debug: ('No shared pages, setting ksm_merge_across_nodes to', 1) 2018-07-31 11:49:44,286 - mom.Evaluator - DEBUG - debug: ('entry: apply_NUMA_policy',) 2018-07-31 11:49:44,286 - mom.Evaluator - DEBUG - debug: (1, '=ksm_merge_across_nodes ACTUAL from kernel') 2018-07-31 11:49:44,286 - mom.Evaluator - DEBUG - debug: (1, '=ksmMergeAcrossNodes REQUIRED from oVirt-engine') 2018-07-31 11:49:44,289 - mom.Evaluator - DEBUG - debug: ('exit: apply_NUMA_policy return_value = ', 0) 2018-07-31 11:49:44,310 - mom.Policy - DEBUG - Results: [0, 1, 1, 1, 0, 1, 1, 0.2, 0.05, 0.2, 0.05, 0.0025, 'change_big_enough', 'shrink_guest', 'grow_guest', 0.22506353015141964, 'balloon_logic', [0], 'guest_qos', [0], 300, -50, 64, 1250, 10, 0.2, 'change_npages', 'apply_NUMA_policy', 6550590.4, 30058208, None, 0, 0, None, -1, 100000, 'check_and_set_quota', 'reset_quota_and_period', [None], 0, 'set_io_limits', 'reset_io_limits', [0]] Does I miss something ? MoM behavior ? Thanks for all !
Hello, Is someone have an idea regarding my Memory balloon misconfig :/ ? Thanks !
Hi, sorry I could not answer yesterday. What you need to check is whether the VM is reported as ready in the log (/var/log/vdsm/mom.log). You should see lines like: 2018-08-01 10:52:25,648 - mom.GuestMonitor.Thread - INFO - GuestMonitor-ms-vhost-1 starting 2018-08-01 10:52:25,648 - mom.GuestManager - DEBUG - added monitor for guest c4f1ea89-457a-4049-97d7-11cf751d00f0 2018-08-01 10:52:25,648 - mom.Monitor - DEBUG - Using optional fields: set(['swap_out', 'swap_usage', 'balloon_cur', 'swap_total', 'balloon_min', 'major_fault', 'swap_in', 'io_tune_current', 'mem_unused', 'balloon_max', 'minor_fault', 'io_tune', 'mem_available']) 2018-08-01 10:52:25,649 - mom.Monitor - DEBUG - Using fields: set(['vcpu_count', 'vcpu_period', 'vcpu_user_limit', 'vcpu_quota']) ... 2018-08-01 10:52:25,808 - mom.Monitor - INFO - ms-vhost-1 is ready There might be a message reporting that some required data fields are not available and the ballooning won't work for that VM in such case. Best regards Martin Sivak On Wed, Aug 1, 2018 at 10:45 AM, <jeanbaptiste@nfrance.com> wrote:
Hello,
Is someone have an idea regarding my Memory balloon misconfig :/ ?
Thanks ! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FTNY4FQ4W52Z62...
Hi Martin, No problem, thanks for our answer ! In posted log, you can see: 2018-07-31 11:49:44,258 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. I think issue start here :/ ? I can't find a INFO log file with My_VM- is ready.... All Centos guests are fully supported ? (Centos 7.X branch) Regards,
Hi,
2018-07-31 11:49:44,258 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring.
No, this is "normal". Should not affect ballooning.
I can't find a INFO log file with My_VM- is ready....
And the line where the GuestMonitor is starting? You need to get back to the time the VM was started or restart MOM (mom-vdsm service).
All Centos guests are fully supported ? (Centos 7.X branch)
Yes I think so. Martin On Wed, Aug 1, 2018 at 2:28 PM, <jeanbaptiste@nfrance.com> wrote:
Hi Martin,
No problem, thanks for our answer ! In posted log, you can see: 2018-07-31 11:49:44,258 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. I think issue start here :/ ? I can't find a INFO log file with My_VM- is ready.... All Centos guests are fully supported ? (Centos 7.X branch)
Regards, _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AWZLKUETHSHTEC...
Hi, Log from my hypervisor after launch the test VM: 2018-08-01 17:45:58,209 - mom.RPCServer - INFO - ping() 2018-08-01 17:45:58,211 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:45:58,346 - mom.RPCServer - INFO - ping() 2018-08-01 17:45:58,347 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:45:58,759 - mom.GuestMonitor.Thread - INFO - GuestMonitor-ovirt-dev-c7-vm1ending 2018-08-01 17:45:58,944 - mom.Controllers.KSM - INFO - Updating KSM configuration: pages_to_scan:0 merge_across_nodes:1 run:0 sleep_millisecs:0 2018-08-01 17:46:13,310 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:13,313 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:13,386 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:13,387 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:28,380 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:28,381 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:28,423 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:28,424 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:28,748 - mom.GuestMonitor.Thread - INFO - GuestMonitor-ovirt-dev-c7-vm1 starting 2018-08-01 17:46:43,402 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:43,403 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:43,459 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:43,460 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:43,772 - mom.Monitor - INFO - ovirt-dev-c7-vm1 is ready 2018-08-01 17:46:58,461 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:58,462 - mom.RPCServer - INFO - getStatistics() 2018-08-01 17:46:58,502 - mom.RPCServer - INFO - ping() 2018-08-01 17:46:58,503 - mom.RPCServer - INFO - getStatistics() After that I lauch my perl program to consume RAM and stop it [root@ovirt-dev-c7-vm1 ~]# free -m total used free shared buff/cache available Mem: 7820 6876 846 2 96 745 Swap: 8063 109 7954 [root@ovirt-dev-c7-vm1 ~]# free -m total used free shared buff/cache available Mem: 7820 188 7535 2 96 7433 Swap: 8063 109 7954 [root@ovirt-dev-c7-vm1 ~]# On hypervisor side, RAM is not released : [root@ovirt-dev1 ~]# free -m total used free shared buff/cache available Mem: 31989 8750 22657 17 581 22816 Swap: 16127 0 16127 [root@ovirt-dev1 ~]# top 22192 vdsm 20 0 681968 24508 5440 S 1,7 0,1 0:01.81 python 22525 qemu 20 0 9126664 8,0g 13664 S 1,7 25,4 0:42.57 qemu-kvm qemu-kvm process still use 8GB RAM instead of ~300MB. An idea ? Thanks !
Hi, did you put the host to maintenance and activated it again when you enabled ballooning? Or search for a slightly hidden Sync MOM policy link somewhere in the host area that would force it. The log seems to indicate ballooning is either disabled or "not necessary". Martin On Tue, Jul 31, 2018 at 12:07 PM, <jeanbaptiste@nfrance.com> wrote:
Hello,
I'm trying to test Memory balloon useful functionality, but l'm unable to make it work successfuly.
Firstly, I enable the MOM on my Cluster: Memory Balloon Enable Memory Balloon Optimization
Secondly, I configure a VM virtual machine like this: On system tab: Memory Size : 28672 MB Maximum memory : 28672 MB On Ressource allocation tab: Memory Allocation: Physical Memory Guaranteed : 2048 MB Memory Balloon Device: Enabled
Thirdly, I test 1- I boot the VM. After boot VM consume 200MB 2- I lauch perl program which consume 26GB. 3- During this, Host RAM usage is near 90% 4- I stop the perl program, and on guest side, RAM usage drop down to 200Mo 5- On host side, RAM usage keep to 24 - 26GB. I wait some time to show if the qemu-vm process drop is host-ram-usage. Ram usage keep to 24GB. 6- I try to launch another VM like the first one (but with 8GB RAM and 2GB Guaranteed) , on the host (to "force" Mom ?) but I'm unable to lauch it, Host free mem is insufficient
I change MoM log to debug to show something: 2018-07-31 11:49:31,887 - mom.RPCServer - INFO - ping() 2018-07-31 11:49:31,888 - mom.RPCServer - INFO - getStatistics() 2018-07-31 11:49:31,888 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring.
2018-07-31 11:49:43,313 - mom.VdsmRpcBase - DEBUG - VM List: [u'9632443f-d302-43f7-a279-778e64ee98f4'] 2018-07-31 11:49:43,465 - mom.VdsmRpcBase - DEBUG - Memory stats: {'swap_out': 0, 'swap_usage': 0, 'mem_free': 28221020, 'major_fault': 0, 'swap_in': 0, 'swap_total': 0, 'mem_available': 28650472, 'minor_fault': 99, 'mem_unused': 28221020} 2018-07-31 11:49:43,468 - mom.Monitor - DEBUG - Collector <mom.Collectors.GuestIoTuneOptional.GuestIoTuneOptional instance at 0x167de18> did not return any data 2018-07-31 11:49:43,931 - mom.RPCServer - INFO - ping() 2018-07-31 11:49:43,932 - mom.RPCServer - INFO - getStatistics() 2018-07-31 11:49:43,932 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. 2018-07-31 11:49:44,258 - mom.Monitor - DEBUG - Field 'mem_free' not known. Ignoring. 2018-07-31 11:49:44,284 - mom.Evaluator - DEBUG - debug: ('No shared pages, setting ksm_merge_across_nodes to', 1) 2018-07-31 11:49:44,286 - mom.Evaluator - DEBUG - debug: ('entry: apply_NUMA_policy',) 2018-07-31 11:49:44,286 - mom.Evaluator - DEBUG - debug: (1, '=ksm_merge_across_nodes ACTUAL from kernel') 2018-07-31 11:49:44,286 - mom.Evaluator - DEBUG - debug: (1, '=ksmMergeAcrossNodes REQUIRED from oVirt-engine') 2018-07-31 11:49:44,289 - mom.Evaluator - DEBUG - debug: ('exit: apply_NUMA_policy return_value = ', 0) 2018-07-31 11:49:44,310 - mom.Policy - DEBUG - Results: [0, 1, 1, 1, 0, 1, 1, 0.2, 0.05, 0.2, 0.05, 0.0025, 'change_big_enough', 'shrink_guest', 'grow_guest', 0.22506353015141964, 'balloon_logic', [0], 'guest_qos', [0], 300, -50, 64, 1250, 10, 0.2, 'change_npages', 'apply_NUMA_policy', 6550590.4, 30058208, None, 0, 0, None, -1, 100000, 'check_and_set_quota', 'reset_quota_and_period', [None], 0, 'set_io_limits', 'reset_io_limits', [0]]
Does I miss something ? MoM behavior ?
Thanks for all ! _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QQ7EIZMCBXHTKO...
Hi, Does I need to activate for Clusters / Hosts, something more than: On my Cluster > Optimization > Memory Balloon > Enable Memory Balloon Optimization I've already reboot / take in maintenance and Force MOM policy several time since I try to make work Balloon How to check if MoM work on my host ? Which logs does I need to have if MoM work as expected ? Thanks
Hi, Initially, when the VM is started, it consumes only 200 MB, because the QEMU process allocates memory from the OS only when accessed by the VM. Once the VM uses 26 GB, QEMU process will also use at least 26 GB. The ballooning starts shrinking the VM, only if the free memory on the host is below a threshold, by default 20%. And it will grow the VM back if the free memory is above the threshold. So once it stabilizes, the host will be using around 80% of memory. If the host has 30 GB of memory, 80% is 24 GB, so the ballooning seems to be working correctly. Regards, Andrej On Thu, 2 Aug 2018 at 12:38, <jeanbaptiste@nfrance.com> wrote:
Hi,
Does I need to activate for Clusters / Hosts, something more than:
On my Cluster > Optimization > Memory Balloon > Enable Memory Balloon Optimization I've already reboot / take in maintenance and Force MOM policy several time since I try to make work Balloon
How to check if MoM work on my host ? Which logs does I need to have if MoM work as expected ?
Thanks _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZPXUAI4YTWCU5D...
Hello, Yes thanks you, MoM work correctly. Indeed 80% used threshold need to be raise for memory release work :) Thanks for support ! Bye,
participants (3)
-
Andrej Krejcir -
jeanbaptiste＠nfrance.com -
Martin Sivak