<p dir="ltr">I think someone created one. Don't know for sure... What do you think it is then? </p>
<div class="gmail_quote">On Feb 21, 2014 11:24 AM, "Martin Sivak" <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi<br>
<br>
I think I found the issue. I suspect it will happen when you get over 35% of used memory.<br>
<br>
Can you please create a bug for this issue? Or is there one already?<br>
<br>
--<br>
Martin Sivák<br>
<a href="mailto:msivak@redhat.com">msivak@redhat.com</a><br>
Red Hat Czech<br>
RHEV-M SLA / Brno, CZ<br>
<br>
----- Original Message -----<br>
> I'm waiting for the next alarm. In the meanwhile this is the output now:<br>
> vdsClient -s saturnus2 getVmStats 3b9aa245-75ff-42e8-b921-1c9ce61826bf<br>
><br>
> 3b9aa245-75ff-42e8-b921-1c9ce61826bf<br>
> Status = Running<br>
> guestFQDN = galatea.***.****<br>
> memUsage = 32<br>
> acpiEnable = true<br>
> netIfaces = [{'inet6': [], 'hw': '52:54:00:49:11:9d', 'inet':<br>
> ['192.168.99.19'], 'name': 'eth1'}, {'inet6': [], 'hw':<br>
> '00:1a:4a:67:c4:b3', 'inet': ['10.110.X.X'], 'name': 'eth2'}, {'inet6': [],<br>
> 'hw': '52:54:00:f1:22:48', 'inet': ['192.168.122.1'], 'name': 'virbr0'}]<br>
> pid = 38144<br>
> session = Unknown<br>
> vmType = kvm<br>
> timeOffset = 0<br>
> balloonInfo = {'balloon_max': '12582912', 'balloon_min': '8388608',<br>
> 'balloon_target': '12582912', 'balloon_cur': '12582912'}<br>
> pauseCode = NOERR<br>
> disksUsage = [{'path': '/', 'total': '37655093248', 'fs': 'ext4',<br>
> 'used': '21765771264'}, {'path': '/boot', 'total': '507744256', 'fs':<br>
> 'ext4', 'used': '49356800'}]<br>
> network = {'vnet0': {'macAddr': '00:1a:4a:67:c4:b3', 'rxDropped': '0',<br>
> 'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0',<br>
> 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': 'vnet0'},<br>
> 'vnet1': {'macAddr': '52:54:00:49:11:9d', 'rxDropped': '0', 'txDropped':<br>
> '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0',<br>
> 'state': 'unknown', 'speed': '1000', 'name': 'vnet1'}, 'vnet2': {'macAddr':<br>
> '52:54:00:66:d3:aa', 'rxDropped': '0', 'txDropped': '0', 'rxErrors': '0',<br>
> 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown',<br>
> 'speed': '1000', 'name': 'vnet2'}}<br>
> memoryStats = {'swap_out': '0', 'majflt': '0', 'mem_free': '8520656',<br>
> 'swap_in': '0', 'pageflt': '396', 'mem_total': '12197520', 'mem_unused':<br>
> '1580896'}<br>
> guestName = <a href="http://galatea.brusselsairport.aero" target="_blank">galatea.brusselsairport.aero</a><br>
> elapsedTime = 2251149<br>
> displayType = qxl<br>
> cpuSys = 12.76<br>
> appsList = ['ovirt-guest-agent-1.0.8-1.el6', 'kernel-2.6.32-71.el6']<br>
> guestOs = 2.6.32-71.el6.x86_64<br>
> username = vandenpt<br>
> hash = -3779086589437073991<br>
> displayIp = 0<br>
> displayPort = 5900<br>
> guestIPs = 192.168.99.19 10.110.50.84 192.168.122.1<br>
> kvmEnable = true<br>
> disks = {'vda': {'readLatency': '0', 'apparentsize': '45097156608',<br>
> 'writeLatency': '1313701', 'imageID':<br>
> '528a377f-4f98-4023-92de-ce52c394d4a9', 'flushLatency': '187524',<br>
> 'readRate': '0.00', 'truesize': '45097156608', 'writeRate': '8502.90'},<br>
> 'hdc': {'readLatency': '0', 'apparentsize': '0', 'writeLatency': '0',<br>
> 'flushLatency': '0', 'readRate': '0.00', 'truesize': '0', 'writeRate':<br>
> '0.00'}}<br>
> monitorResponse = 0<br>
> statsAge = 0.47<br>
> cpuUser = 68.45<br>
> lastLogin = 1392707812.29<br>
> clientIp =<br>
> displaySecurePort = 5901<br>
><br>
><br>
><br>
> 2014-02-17 14:00 GMT+01:00 Martin Sivak <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>>:<br>
><br>
> > Hi Koen,<br>
> ><br>
> > can you try the mentioned vdsClient command (vdsClient getVmStats<br>
> > <affected vm>) as soon as you get the warning about negative memory? The<br>
> > one David sent to us does not contain the negative number and we weren't<br>
> > able to find the issue based on it.<br>
> ><br>
> > Thanks<br>
> ><br>
> > --<br>
> > Martin Sivák<br>
> > <a href="mailto:msivak@redhat.com">msivak@redhat.com</a><br>
> > Red Hat Czech<br>
> > RHEV-M SLA / Brno, CZ<br>
> ><br>
> > ----- Original Message -----<br>
> > > Any updates about the issue? Because just now, the alarm went off again<br>
> > :-)<br>
> > > Kind regards,<br>
> > ><br>
> > > Koen<br>
> > ><br>
> > ><br>
> > > 2014-02-13 16:55 GMT+01:00 Martin Sivak <<a href="mailto:msivak@redhat.com">msivak@redhat.com</a>>:<br>
> > ><br>
> > > > Thank you.<br>
> > > ><br>
> > > > I wonder why mem_total is smaller than balloon_cur (and max). I will<br>
> > have<br>
> > > > to dig a bit more to see what happened.<br>
> > > ><br>
> > > > --<br>
> > > > Martin Sivák<br>
> > > > <a href="mailto:msivak@redhat.com">msivak@redhat.com</a><br>
> > > > Red Hat Czech<br>
> > > > RHEV-M SLA / Brno, CZ<br>
> > > ><br>
> > > > ----- Original Message -----<br>
> > > > > ---------- Forwarded message ----------<br>
> > > > > From: "david van zeebroeck" < <a href="mailto:david@analytics.brusselsairport.be">david@analytics.brusselsairport.be</a> ><br>
> > > > > Date: Feb 13, 2014 4:31 PM<br>
> > > > > Subject: Re: [Users] Memory usage<br>
> > > > > To: "Martin Sivak" < <a href="mailto:msivak@redhat.com">msivak@redhat.com</a> ><br>
> > > > > Cc: "Doron Fediuck" < <a href="mailto:dfediuck@redhat.com">dfediuck@redhat.com</a> >, "Koen Vanoppen" <<br>
> > > > > <a href="mailto:vanoppen.koen@gmail.com">vanoppen.koen@gmail.com</a> >, < <a href="mailto:users@ovirt.org">users@ovirt.org</a> ><br>
> > > > ><br>
> > > > > in attachement is the output of vdsclient<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > On Thu, Feb 13, 2014 at 2:26 PM, Martin Sivak < <a href="mailto:msivak@redhat.com">msivak@redhat.com</a> ><br>
> > > > wrote:<br>
> > > > ><br>
> > > > ><br>
> > > > > Hi everybody,<br>
> > > > ><br>
> > > > > would it be possible to add the output of vdsClient getVmStats<br>
> > <affected<br>
> > > > vm<br>
> > > > > id> from the affected host here and to the bug?<br>
> > > > ><br>
> > > > > Also can we please get the bug number that tracks this issue? I<br>
> > could not<br>
> > > > > find it in BZ.<br>
> > > > ><br>
> > > > > --<br>
> > > > > Martin Sivák<br>
> > > > > <a href="mailto:msivak@redhat.com">msivak@redhat.com</a><br>
> > > > > Red Hat Czech<br>
> > > > > RHEV-M SLA / Brno, CZ<br>
> > > > ><br>
> > > > > ----- Original Message -----<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > ----- Original Message -----<br>
> > > > > > > From: "Koen Vanoppen" < <a href="mailto:vanoppen.koen@gmail.com">vanoppen.koen@gmail.com</a> ><br>
> > > > > > > To: <a href="mailto:users@ovirt.org">users@ovirt.org</a><br>
> > > > > > > Sent: Thursday, February 13, 2014 12:18:24 PM<br>
> > > > > > > Subject: Re: [Users] Memory usage<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > We can't turn off the memory balloon option because we are<br>
> > running<br>
> > > > the<br>
> > > > > > > 3.3.2<br>
> > > > > > > with the bug of the memory balloon. Thx for the help!<br>
> > > > > ><br>
> > > > > > Which balloon bug?<br>
> > > > > ><br>
> > > > > > > On Feb 13, 2014 10:51 AM, "Doron Fediuck" < <a href="mailto:dfediuck@redhat.com">dfediuck@redhat.com</a>><br>
> > > > wrote:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > ----- Original Message -----<br>
> > > > > > > > From: "René Koch" < <a href="mailto:rkoch@linuxland.at">rkoch@linuxland.at</a> ><br>
> > > > > > > > To: "Doron Fediuck" < <a href="mailto:dfediuck@redhat.com">dfediuck@redhat.com</a> ><br>
> > > > > > > > Cc: "Koen Vanoppen" < <a href="mailto:vanoppen.koen@gmail.com">vanoppen.koen@gmail.com</a> >,<br>
> > <a href="mailto:users@ovirt.org">users@ovirt.org</a> ,<br>
> > > > > > > > "Martin<br>
> > > > > > > > Sivak" < <a href="mailto:msivak@redhat.com">msivak@redhat.com</a> ><br>
> > > > > > > > Sent: Thursday, February 13, 2014 11:16:27 AM<br>
> > > > > > > > Subject: Re: [Users] Memory usage<br>
> > > > > > > ><br>
> > > > > > > > On Thu, 2014-02-13 at 03:49 -0500, Doron Fediuck wrote:<br>
> > > > > > > > ><br>
> > > > > > > > > ----- Original Message -----<br>
> > > > > > > > > > From: "René Koch" < <a href="mailto:rkoch@linuxland.at">rkoch@linuxland.at</a> ><br>
> > > > > > > > > > To: "Doron Fediuck" < <a href="mailto:dfediuck@redhat.com">dfediuck@redhat.com</a> ><br>
> > > > > > > > > > Cc: "Koen Vanoppen" < <a href="mailto:vanoppen.koen@gmail.com">vanoppen.koen@gmail.com</a> >,<br>
> > > > <a href="mailto:users@ovirt.org">users@ovirt.org</a> ,<br>
> > > > > > > > > > "Martin<br>
> > > > > > > > > > Sivak" < <a href="mailto:msivak@redhat.com">msivak@redhat.com</a> ><br>
> > > > > > > > > > Sent: Wednesday, February 12, 2014 7:31:47 PM<br>
> > > > > > > > > > Subject: Re: [Users] Memory usage<br>
> > > > > > > > > ><br>
> > > > > > > > > > On Wed, 2014-02-12 at 11:22 -0500, Doron Fediuck wrote:<br>
> > > > > > > > > > ><br>
> > > > > > > > > > > ----- Original Message -----<br>
> > > > > > > > > > > > From: "René Koch" < <a href="mailto:rkoch@linuxland.at">rkoch@linuxland.at</a> ><br>
> > > > > > > > > > > > To: "Koen Vanoppen" < <a href="mailto:vanoppen.koen@gmail.com">vanoppen.koen@gmail.com</a> ><br>
> > > > > > > > > > > > Cc: <a href="mailto:users@ovirt.org">users@ovirt.org</a><br>
> > > > > > > > > > > > Sent: Wednesday, February 12, 2014 4:18:37 PM<br>
> > > > > > > > > > > > Subject: Re: [Users] Memory usage<br>
> > > > > > > > > > > ><br>
> > > > > > > > > > > > On Wed, 2014-02-12 at 15:14 +0100, Koen Vanoppen wrote:<br>
> > > > > > > > > > > > > In The GUI, it says it's using 25% of the memory.<br>
> > > > > > > > > > > ><br>
> > > > > > > > > > > ><br>
> > > > > > > > > > > > I guess it's the real value, right?<br>
> > > > > > > > > > > > The same happened for the memcached vm, someone<br>
> > reported<br>
> > > > to me<br>
> > > > > > > > > > > > -<br>
> > > > > > > > > > > > negative value in REST-API, but correct graph in oVirt<br>
> > > > webadmin<br>
> > > > > > > > > > > > GUI.<br>
> > > > > > > > > > > ><br>
> > > > > > > > > > > > I fear I have no idea how this can happen - so maybe<br>
> > > > someone<br>
> > > > > > > > > > > > else<br>
> > > > > > > > > > > > can<br>
> > > > > > > > > > > > help you troubleshoot this issue.<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > 2014-02-12 15:10 GMT+01:00 Koen Vanoppen <<br>
> > > > > > > > > > > > > <a href="mailto:vanoppen.koen@gmail.com">vanoppen.koen@gmail.com</a> >:<br>
> > > > > > > > > > > > > Thanks for the quick respons, but there is no<br>
> > memcached<br>
> > > > > > > > > > > > > running on that VM.<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > Kind regards<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > 2014-02-12 15:06 GMT+01:00 René Koch <<br>
> > > > <a href="mailto:rkoch@linuxland.at">rkoch@linuxland.at</a> >:<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > On Wed, 2014-02-12 at 14:55 +0100, Koen Vanoppen<br>
> > > > > > > > > > > > > wrote:<br>
> > > > > > > > > > > > > > Dear all,<br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > > When we monitor one of our machines, we noticed<br>
> > > > > > > > > > > > > > that<br>
> > > > > > > > > > > > > there was one vm<br>
> > > > > > > > > > > > > > that was constantly giving a error of memory<br>
> > > > > > > > > > > > > > usage.<br>
> > > > > > > > > > > > > But when we took a<br>
> > > > > > > > > > > > > > look at it, there is actually nothing wrong with<br>
> > > > > > > > > > > > > > it.<br>
> > > > > > > > > > > > > Now we looked<br>
> > > > > > > > > > > > > > furhter then that. We looked at the API of the<br>
> > > > > > > > > > > > > machine and noticed<br>
> > > > > > > > > > > > > > something very strange:<br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > > <statistic<br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > ><br>
> > href="/api/vms/3b9aa245-75ff-42e8-b921-1c9ce61826bf/statistics/b7499508-c1c3-32f0-8174-c1783e57bb08"<br>
> > > > > > > > > > > > ><br>
> > > ><br>
> > id="b7499508-c1c3-32f0-8174-c1783e57bb08"><name>memory.used</name><description>Memory<br>
> > > > > > > > > > > > > used (agent)</description><values<br>
> > > > > > > > > > > > ><br>
> > > ><br>
> > type="INTEGER"><value><datum>-944892806</datum></value></values><type>GAUGE</type><unit>BYTES</unit><br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > > It's a negative...<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > Do you have memcached running in this vm?<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > I heard about this issue with memcached, but never<br>
> > > > > > > > > > > > > tested memcached in<br>
> > > > > > > > > > > > > my oVirt environment. You get the real usage value<br>
> > > > > > > > > > > > > with<br>
> > > > > > > > > > > > > memory.used = memory.installed + memory.used<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > Regards,<br>
> > > > > > > > > > > > > René<br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > > What could be the problem?<br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > > Kind regards,<br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > > > koen<br>
> > > > > > > > > > > > > ><br>
> > > > > > > > > > > > ><br>
> > > > > > > > > > ><br>
> > > > > > > > > > > Guys,<br>
> > > > > > > > > > > these values are usually a result of overcommitment<br>
> > mechanism<br>
> > > > > > > > > > > usage.<br>
> > > > > > > > > > > For example, if KSM is effective, it will free a lot of<br>
> > > > memory<br>
> > > > > > > > > > > pages,<br>
> > > > > > > > > > > and total-free-committed becomes negative.<br>
> > > > > > > > > ><br>
> > > > > > > > > ><br>
> > > > > > > > > > Thanks a lot for the information.<br>
> > > > > > > > > ><br>
> > > > > > > > > ><br>
> > > > > > > > > > ><br>
> > > > > > > > > > > This was reported in<br>
> > > > > > > > > > > <a href="https://bugzilla.redhat.com/show_bug.cgi?id=977758" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=977758</a><br>
> > > > > > > > > > > and the engine is using memFree reported by vdsm, which<br>
> > is<br>
> > > > more<br>
> > > > > > > > > > > accurate.<br>
> > > > > > > > > > ><br>
> > > > > > > > > > > The API reports the old version due to backwards<br>
> > > > compatibility.<br>
> > > > > > > > > ><br>
> > > > > > > > > ><br>
> > > > > > > > > > I just had a look at the bugzilla report and the RHEV<br>
> > > > documentation<br>
> > > > > > > > > > which says "Current memory in bytes used by the virtual<br>
> > > > machine.".<br>
> > > > > > > > > > So this means, the reported values are totally useless for<br>
> > > > > > > > > > monitoring<br>
> > > > > > > > > > memory usage of a virtual machine if KSM is active. I would<br>
> > > > expect<br>
> > > > > > > > > > to<br>
> > > > > > > > > > get the memory usage of a virtual machine and not how much<br>
> > > > memory<br>
> > > > > > > > > > is<br>
> > > > > > > > > > consumed on the hypervisor (this is pretty useless<br>
> > information<br>
> > > > for<br>
> > > > > > > > > > me).<br>
> > > > > > > > > ><br>
> > > > > > > > > > Is it planned to report the memory usage in a virtual<br>
> > machine<br>
> > > > in<br>
> > > > > > > > > > the<br>
> > > > > > > > > > API<br>
> > > > > > > > > > as well?<br>
> > > > > > > > > ><br>
> > > > > > > > > ><br>
> > > > > > > > > > Regards,<br>
> > > > > > > > > > René<br>
> > > > > > > > > ><br>
> > > > > > > > > ><br>
> > > > > > > > ><br>
> > > > > > > > > René,<br>
> > > > > > > > > started digging into this, but it may take some time.<br>
> > > > > > > > ><br>
> > > > > > > > > Note that there are 2 memory usage reports; one for the host,<br>
> > > > > > > > > and another for each VM.<br>
> > > > > > > > > My response is related to the host, while the original<br>
> > question<br>
> > > > was<br>
> > > > > > > > > for the VM. I still think the root cause is the same (ie-<br>
> > > > related to<br>
> > > > > > > > > overcommitment; for example the VM may have a balloon<br>
> > inflated),<br>
> > > > but<br>
> > > > > > > > > would like to properly check and fix if needed.<br>
> > > > > > > ><br>
> > > > > > > ><br>
> > > > > > > > Yes, I know. So my question was if we can add another value<br>
> > for the<br>
> > > > > > > > usage in the vm in addition to the usage of the vm on the host.<br>
> > > > > > > ><br>
> > > > > > > We'll know once we understand the cause- ie is it a bug needs<br>
> > fixing<br>
> > > > or<br>
> > > > > > > should we indeed add another value<br>
> > > > > > ><br>
> > > > > > > If you have a setup with this issue, try removing the balloon<br>
> > device<br>
> > > > > > > and see if it helps / changes.<br>
> > > > > > ><br>
> > > > > > > ><br>
> > > > > > > > ><br>
> > > > > > > > > In order to make sure we keep track of it, please open a bug<br>
> > > > with all<br>
> > > > > > > > > the<br>
> > > > > > > > > relevant info.<br>
> > > > > > > > ><br>
> > > > > > > ><br>
> > > > > > > > I'll open a bug for this.<br>
> > > > > > > ><br>
> > > > > > > Thanks<br>
> > > > > > ><br>
> > > > > > > ><br>
> > > > > > > > > Thanks and keep reporting!<br>
> > > > > > > > > Doron<br>
> > > > > > > ><br>
> > > > > > > ><br>
> > > > > > > ><br>
> > > > > > ><br>
> > > > > > > _______________________________________________<br>
> > > > > > > Users mailing list<br>
> > > > > > > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> > > > > > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > > > > > ><br>
> > > > > > _______________________________________________<br>
> > > > > > Users mailing list<br>
> > > > > > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> > > > > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > > > > ><br>
> > > > > _______________________________________________<br>
> > > > > Users mailing list<br>
> > > > > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> > > > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > > > ><br>
> > > > ><br>
> > > > > _______________________________________________<br>
> > > > > Users mailing list<br>
> > > > > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> > > > > <a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
> > > > ><br>
> > > ><br>
> > ><br>
> ><br>
><br>
</blockquote></div>