--_000_AM3PR06MB12021070EF302BE6210F1A65828C0AM3PR06MB1202eurp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, I'm trying to track down why my Foreman Centos7 VM continuously crashes wit= h out of Memory errors on my ovirt-3.6 hosts and what seems to happen is th= e following: 1. The Foreman VM only uses 1GB at the beginning and MOM over time reduces = the memory size from 8GB Ram to the defined minimum of 2GB mom.log: 2016-03-18 12:25:42,669 - mom.Controllers.Balloon - INFO - Ballooning guest= :qvie-foreman from 2210584 to 2100054 2. When the VM needs RAM very quickly like if you do an 'cp /dev/zero /run/= zero' the ovirt guest agent crashes with an out of memory error. ovirt-guest-agent.log: Traceback (most recent call last): File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 239, in doWo= rk self.sendUserInfo() File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 355, in send= UserInfo cur_user =3D self.dr.getActiveUser() File "/usr/share/ovirt-guest-agent/GuestAgentLinux2.py", line 325, in get= ActiveUser users =3D os.popen('/usr/bin/users').read().split() OSError: [Errno 12] Cannot allocate memory 3. Resulting in an MOM warning that it can't get any memory stats from the = VM. Leaving the memory size at its minimum size:. mom.log: mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-= guest-agent is not active Any chances to prevent that from happening? Or is this a bug in MOM / ovirt= -guest-agent that should be fixed? Greetings Andreas --_000_AM3PR06MB12021070EF302BE6210F1A65828C0AM3PR06MB1202eurp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> </head> <body dir=3D"ltr"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;back= ground-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"> <p>Hi,</p> <p><br> </p> <p>I'm trying to track down why my Foreman Centos7 VM continuously crashes = with out of Memory errors on my ovirt-3.6 hosts and what seems to happen is= the following:</p> <p><br> </p> <p>1. The Foreman VM only uses 1GB at the beginning and MOM over time reduc= es the memory size from 8GB Ram to the defined minimum of 2GB <br> </p> <p><br> </p> <p>mom.log:<br> 2016-03-18 12:25:42,669 - mom.Controllers.Balloon - INFO - Ballooning guest= :qvie-foreman from 2210584 to 2100054</p> <p><br> </p> <p>2. When the VM needs RAM very quickly like if you do an 'cp /dev/zero /r= un/zero' the ovirt guest agent crashes with an out of memory error.</p> <p><br> </p> <p>ovirt-guest-agent.log:</p> <p>Traceback (most recent call last):<br> File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", li= ne 239, in doWork<br> self.sendUserInfo()<br> File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", li= ne 355, in sendUserInfo<br> cur_user =3D self.dr.getActiveUser()<br> File "/usr/share/ovirt-guest-agent/GuestAgentLinux2.py", l= ine 325, in getActiveUser<br> users =3D os.popen('/usr/bin/users').read().split()<br> OSError: [Errno 12] Cannot allocate memory<br> <br> 3. Resulting in an MOM warning that it can't get any memory stats from the = VM. Leaving the memory size at its minimum size:.<br> </p> <p><br> </p> <p>mom.log:<br> </p> <p>mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovi= rt-guest-agent is not active</p> <p><br> </p> <p>Any chances to prevent that from happening? Or is this a bug in MOM / ov= irt-guest-agent that should be fixed?<br> </p> <p><br> </p> <p>Greetings</p> <p>Andreas<br> <br> </p> </div> </body> </html> --_000_AM3PR06MB12021070EF302BE6210F1A65828C0AM3PR06MB1202eurp_--
Hi, it is not really a bug. You should give the VM more memory if you expect that it will need it that fast. The ballooning mechanism has its disadvantages and this is one of those. We do not react fast enough when such a fast allocation happens. There is really not much we can do from the mom / guest agent side. We could slowly increase the available memory when the guest agent dies (pretty easy to implement) and we might inform the system that a guest agent is an important process (if that is even possible) so OOM killer will avoid it. But that would not help with Cannot allocate memory error as even the OOM killer takes time before it reacts. Using low guaranteed memory values with ballooning for critical VMs is probably not the best idea. Regards -- Martin Sivak msivak@redhat.com SLA / oVirt On Fri, Mar 18, 2016 at 1:00 PM, Andreas Trawöger <andreas.trawoeger@qitasc.com> wrote:
Hi,
I'm trying to track down why my Foreman Centos7 VM continuously crashes with out of Memory errors on my ovirt-3.6 hosts and what seems to happen is the following:
1. The Foreman VM only uses 1GB at the beginning and MOM over time reduces the memory size from 8GB Ram to the defined minimum of 2GB
mom.log: 2016-03-18 12:25:42,669 - mom.Controllers.Balloon - INFO - Ballooning guest:qvie-foreman from 2210584 to 2100054
2. When the VM needs RAM very quickly like if you do an 'cp /dev/zero /run/zero' the ovirt guest agent crashes with an out of memory error.
ovirt-guest-agent.log:
Traceback (most recent call last): File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 239, in doWork self.sendUserInfo() File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 355, in sendUserInfo cur_user = self.dr.getActiveUser() File "/usr/share/ovirt-guest-agent/GuestAgentLinux2.py", line 325, in getActiveUser users = os.popen('/usr/bin/users').read().split() OSError: [Errno 12] Cannot allocate memory
3. Resulting in an MOM warning that it can't get any memory stats from the VM. Leaving the memory size at its minimum size:.
mom.log:
mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-guest-agent is not active
Any chances to prevent that from happening? Or is this a bug in MOM / ovirt-guest-agent that should be fixed?
Greetings
Andreas
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Martin, Whats extremely prone to errors is the way the guest agent checks the active user in GuestAgentLinux2.py line 325: users = os.popen('/usr/bin/users').read().split() This creates a new subprocess every time it is executed and if that doesn't work for whatever reason the ovirt guest agent crashes instantly. But I probably should make that suggestions on the devel list. Regards ________________________________________ Von: Martin Sivak <msivak@redhat.com> Gesendet: Freitag, 18. März 2016 14:44 An: Andreas Trawöger Cc: oVirt Userlist Betreff: Re: [ovirt-users] MOM crashes ovirt-guest-agent Hi, it is not really a bug. You should give the VM more memory if you expect that it will need it that fast. The ballooning mechanism has its disadvantages and this is one of those. We do not react fast enough when such a fast allocation happens. There is really not much we can do from the mom / guest agent side. We could slowly increase the available memory when the guest agent dies (pretty easy to implement) and we might inform the system that a guest agent is an important process (if that is even possible) so OOM killer will avoid it. But that would not help with Cannot allocate memory error as even the OOM killer takes time before it reacts. Using low guaranteed memory values with ballooning for critical VMs is probably not the best idea. Regards -- Martin Sivak msivak@redhat.com SLA / oVirt On Fri, Mar 18, 2016 at 1:00 PM, Andreas Trawöger <andreas.trawoeger@qitasc.com> wrote:
Hi,
I'm trying to track down why my Foreman Centos7 VM continuously crashes with out of Memory errors on my ovirt-3.6 hosts and what seems to happen is the following:
1. The Foreman VM only uses 1GB at the beginning and MOM over time reduces the memory size from 8GB Ram to the defined minimum of 2GB
mom.log: 2016-03-18 12:25:42,669 - mom.Controllers.Balloon - INFO - Ballooning guest:qvie-foreman from 2210584 to 2100054
2. When the VM needs RAM very quickly like if you do an 'cp /dev/zero /run/zero' the ovirt guest agent crashes with an out of memory error.
ovirt-guest-agent.log:
Traceback (most recent call last): File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 239, in doWork self.sendUserInfo() File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 355, in sendUserInfo cur_user = self.dr.getActiveUser() File "/usr/share/ovirt-guest-agent/GuestAgentLinux2.py", line 325, in getActiveUser users = os.popen('/usr/bin/users').read().split() OSError: [Errno 12] Cannot allocate memory
3. Resulting in an MOM warning that it can't get any memory stats from the VM. Leaving the memory size at its minimum size:.
mom.log:
mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-guest-agent is not active
Any chances to prevent that from happening? Or is this a bug in MOM / ovirt-guest-agent that should be fixed?
Greetings
Andreas
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Whats extremely prone to errors is the way the guest agent checks the active user in GuestAgentLinux2.py line 325:
users = os.popen('/usr/bin/users').read().split()
That is something we could probably improve. The maintainer is Vinzenz Feenstra. Martin On Fri, Mar 18, 2016 at 3:13 PM, Andreas Trawöger <andreas.trawoeger@qitasc.com> wrote:
Hi Martin,
Whats extremely prone to errors is the way the guest agent checks the active user in GuestAgentLinux2.py line 325:
users = os.popen('/usr/bin/users').read().split()
This creates a new subprocess every time it is executed and if that doesn't work for whatever reason the ovirt guest agent crashes instantly.
But I probably should make that suggestions on the devel list.
Regards
________________________________________ Von: Martin Sivak <msivak@redhat.com> Gesendet: Freitag, 18. März 2016 14:44 An: Andreas Trawöger Cc: oVirt Userlist Betreff: Re: [ovirt-users] MOM crashes ovirt-guest-agent
Hi,
it is not really a bug. You should give the VM more memory if you expect that it will need it that fast. The ballooning mechanism has its disadvantages and this is one of those. We do not react fast enough when such a fast allocation happens.
There is really not much we can do from the mom / guest agent side. We could slowly increase the available memory when the guest agent dies (pretty easy to implement) and we might inform the system that a guest agent is an important process (if that is even possible) so OOM killer will avoid it. But that would not help with Cannot allocate memory error as even the OOM killer takes time before it reacts.
Using low guaranteed memory values with ballooning for critical VMs is probably not the best idea.
Regards
-- Martin Sivak msivak@redhat.com SLA / oVirt
On Fri, Mar 18, 2016 at 1:00 PM, Andreas Trawöger <andreas.trawoeger@qitasc.com> wrote:
Hi,
I'm trying to track down why my Foreman Centos7 VM continuously crashes with out of Memory errors on my ovirt-3.6 hosts and what seems to happen is the following:
1. The Foreman VM only uses 1GB at the beginning and MOM over time reduces the memory size from 8GB Ram to the defined minimum of 2GB
mom.log: 2016-03-18 12:25:42,669 - mom.Controllers.Balloon - INFO - Ballooning guest:qvie-foreman from 2210584 to 2100054
2. When the VM needs RAM very quickly like if you do an 'cp /dev/zero /run/zero' the ovirt guest agent crashes with an out of memory error.
ovirt-guest-agent.log:
Traceback (most recent call last): File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 239, in doWork self.sendUserInfo() File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 355, in sendUserInfo cur_user = self.dr.getActiveUser() File "/usr/share/ovirt-guest-agent/GuestAgentLinux2.py", line 325, in getActiveUser users = os.popen('/usr/bin/users').read().split() OSError: [Errno 12] Cannot allocate memory
3. Resulting in an MOM warning that it can't get any memory stats from the VM. Leaving the memory size at its minimum size:.
mom.log:
mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-guest-agent is not active
Any chances to prevent that from happening? Or is this a bug in MOM / ovirt-guest-agent that should be fixed?
Greetings
Andreas
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Can you please open a bug? https://bugzilla.redhat.com/enter_bug.cgi?product=ovirt-guest-agent Yaniv Dary Technical Product Manager Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109 Tel : +972 (9) 7692306 8272306 Email: ydary@redhat.com IRC : ydary On Fri, Mar 18, 2016 at 6:20 PM, Martin Sivak <msivak@redhat.com> wrote:
Whats extremely prone to errors is the way the guest agent checks the active user in GuestAgentLinux2.py line 325:
users = os.popen('/usr/bin/users').read().split()
That is something we could probably improve. The maintainer is Vinzenz Feenstra.
Martin
Hi Martin,
Whats extremely prone to errors is the way the guest agent checks the active user in GuestAgentLinux2.py line 325:
users = os.popen('/usr/bin/users').read().split()
This creates a new subprocess every time it is executed and if that doesn't work for whatever reason the ovirt guest agent crashes instantly.
But I probably should make that suggestions on the devel list.
Regards
________________________________________ Von: Martin Sivak <msivak@redhat.com> Gesendet: Freitag, 18. März 2016 14:44 An: Andreas Trawöger Cc: oVirt Userlist Betreff: Re: [ovirt-users] MOM crashes ovirt-guest-agent
Hi,
it is not really a bug. You should give the VM more memory if you expect that it will need it that fast. The ballooning mechanism has its disadvantages and this is one of those. We do not react fast enough when such a fast allocation happens.
There is really not much we can do from the mom / guest agent side. We could slowly increase the available memory when the guest agent dies (pretty easy to implement) and we might inform the system that a guest agent is an important process (if that is even possible) so OOM killer will avoid it. But that would not help with Cannot allocate memory error as even the OOM killer takes time before it reacts.
Using low guaranteed memory values with ballooning for critical VMs is probably not the best idea.
Regards
-- Martin Sivak msivak@redhat.com SLA / oVirt
On Fri, Mar 18, 2016 at 1:00 PM, Andreas Trawöger <andreas.trawoeger@qitasc.com> wrote:
Hi,
I'm trying to track down why my Foreman Centos7 VM continuously crashes with out of Memory errors on my ovirt-3.6 hosts and what seems to happen is
following:
1. The Foreman VM only uses 1GB at the beginning and MOM over time reduces the memory size from 8GB Ram to the defined minimum of 2GB
mom.log: 2016-03-18 12:25:42,669 - mom.Controllers.Balloon - INFO - Ballooning guest:qvie-foreman from 2210584 to 2100054
2. When the VM needs RAM very quickly like if you do an 'cp /dev/zero /run/zero' the ovirt guest agent crashes with an out of memory error.
ovirt-guest-agent.log:
Traceback (most recent call last): File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 239, in doWork self.sendUserInfo() File "/usr/share/ovirt-guest-agent/OVirtAgentLogic.py", line 355, in sendUserInfo cur_user = self.dr.getActiveUser() File "/usr/share/ovirt-guest-agent/GuestAgentLinux2.py", line 325, in getActiveUser users = os.popen('/usr/bin/users').read().split() OSError: [Errno 12] Cannot allocate memory
3. Resulting in an MOM warning that it can't get any memory stats from
On Fri, Mar 18, 2016 at 3:13 PM, Andreas Trawöger <andreas.trawoeger@qitasc.com> wrote: the the
VM. Leaving the memory size at its minimum size:.
mom.log:
mom.Collectors.GuestMemory - WARNING - getVmMemoryStats() error: The ovirt-guest-agent is not active
Any chances to prevent that from happening? Or is this a bug in MOM / ovirt-guest-agent that should be fixed?
Greetings
Andreas
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Andreas Trawöger -
Martin Sivak -
Yaniv Dary