From: "Jason Keltz" <jason.keltz(a)gmail.com&gt; To: "users" <users(a)ovirt.org&gt; Cc: "Eli Mesika" <emesika(a)redhat.com&gt; Sent: Monday, June 15, 2015 11:08:35 PM Subject: problems with power management using idrac7 on r620 Hi. I've been having problem with power management using iDRAC 7 EXPRESS on a Dell R620. This uses a shared LOM as opposed to Enterprise that has a dedicated one. Every now and then, idrac simply stops responding to ping, so it can't respond to status commands from the proxy. If I send a reboot with "ipmitool mc reset cold" command, the idrac reboots and comes back, but after the problem has occurred, even after a reboot, it responds to ping, but drops 80+% of packets. The only way I can "solve" the problem is to physically restart the server. This isn't just happening on one R620 - it's happening on all of my ovirt hosts. I highly suspect it has to do with a memory leak, and being monitored by engine causes the problem. I had applied a recent firmware upgrade that was supposed to "solve" this kind of problem, but it doesn't. In other to provide Dell with more details, can someone tell me how often each host is being queried for status? I can't seem to find that info. The idrac on my file server doesn't seem to exhibit the same problem, and I suspect that is because it isn't being queried. Thanks, Jason.

On 06/16/2015 09:37 AM, Eli Mesika wrote: > CCing Marek Grac > > ----- Original Message ----- > >> From: "Jason Keltz" <jason.keltz(a)gmail.com&gt; >> To: "users" <users(a)ovirt.org&gt; >> Cc: "Eli Mesika" <emesika(a)redhat.com&gt; >> Sent: Monday, June 15, 2015 11:08:35 PM >> Subject: problems with power management using idrac7 on r620 >> >> Hi. >> >> I've been having problem with power management using iDRAC 7 EXPRESS on >> a Dell R620. This uses a shared LOM as opposed to Enterprise that has a >> dedicated one. Every now and then, idrac simply stops responding to >> ping, so it can't respond to status commands from the proxy. If I send >> a reboot with "ipmitool mc reset cold" command, the idrac reboots and >> comes back, but after the problem has occurred, even after a reboot, it >> responds to ping, but drops 80+% of packets. The only way I can "solve" >> the problem is to physically restart the server. This isn't just >> happening on one R620 - it's happening on all of my ovirt hosts. I >> highly suspect it has to do with a memory leak, and being monitored by >> engine causes the problem. I had applied a recent firmware upgrade >> that was supposed to "solve" this kind of problem, but it doesn't. In >> other to provide Dell with more details, can someone tell me how often >> each host is being queried for status? I can't seem to find that info. >> The idrac on my file server doesn't seem to exhibit the same problem, >> and I suspect that is because it isn't being queried. >> > Hi, fence agent for IPMI is based on ipmitool. So if ping/ipmitool is not working there is not much to do about it. I don't know enough about oVirt engine but there is no real place where fence agent can memory leak because it does not run as daemon. m,

----- Original Message ----- > From: "Jason Keltz" <jason.keltz(a)gmail.com&gt; > To: "Marek marx Grac" <mgrac(a)redhat.com&gt; > Cc: "Eli Mesika" <emesika(a)redhat.com&gt;, "users" <users(a)ovirt.org&gt; > Sent: Wednesday, June 17, 2015 12:02:48 PM > Subject: Re: problems with power management using idrac7 on r620 > > Hi Marek. > > Actually its the idrac that I believe has the memory leak. Dell wants to > know how often ovirt is querying the idrac for status and whether the delay > is configurable. Well oVirt does not query the status automatically by default There is a feature that enables that http://www.ovirt.org/Features/PMHealthCheck Basically this feature depends on 2 configuration values : PMHealthCheckEnabled that shoul be true if the feature is enabled PMHealthCheckIntervalInSec which is defaulted to 3600 Sec , so it is checked in that case once in an hour So, first please check if this is enabled in your environment engine-config -g PMHealthCheckEnabled engine-config -g PMHealthCheckIntervalInSec Other scenario when status is used is when host becomes non-responsive In case that host become non responsive : After a grace period that depends on the host load and if it is SPM or not a soft-fence attempt (vdsmd service restart) is issued If the soft-fence attempt fails we will do a real fencing (if power management is configured correctly on the host and a proxy host is found) We are sending a STOP command We are sending by default 18 status command , one each 10 sec until we get 'off' status from the agent We are sending a START command We are sending by default 18 status command , one each 10 sec until we get 'on' status from the agent Those depends on the following configuration variables : FenceStopStatusRetries - default 18 FenceStopStatusDelayBetweenRetriesInSec - default 10 FenceStartStatusRetries - default 18 FenceStartStatusDelayBetweenRetriesInSec - default 10 This can be changed using the engine-config tool (requires restart to take affect) > > Jason. > On Jun 17, 2015 2:42 AM, "Marek "marx" Grac" <mgrac(a)redhat.com&gt; wrote: > > > > > > > On 06/16/2015 09:37 AM, Eli Mesika wrote: > > > >> CCing Marek Grac > >> > >> ----- Original Message ----- > >> > >>> From: "Jason Keltz" <jason.keltz(a)gmail.com&gt; > >>> To: "users" <users(a)ovirt.org&gt; > >>> Cc: "Eli Mesika" <emesika(a)redhat.com&gt; > >>> Sent: Monday, June 15, 2015 11:08:35 PM > >>> Subject: problems with power management using idrac7 on r620 > >>> > >>> Hi. > >>> > >>> I've been having problem with power management using iDRAC 7 EXPRESS on > >>> a Dell R620. This uses a shared LOM as opposed to Enterprise that has a > >>> dedicated one. Every now and then, idrac simply stops responding to > >>> ping, so it can't respond to status commands from the proxy. If I send > >>> a reboot with "ipmitool mc reset cold" command, the idrac reboots and > >>> comes back, but after the problem has occurred, even after a reboot, it > >>> responds to ping, but drops 80+% of packets. The only way I can "solve" > >>> the problem is to physically restart the server. This isn't just > >>> happening on one R620 - it's happening on all of my ovirt hosts. I > >>> highly suspect it has to do with a memory leak, and being monitored by > >>> engine causes the problem. I had applied a recent firmware upgrade > >>> that was supposed to "solve" this kind of problem, but it doesn't. In > >>> other to provide Dell with more details, can someone tell me how often > >>> each host is being queried for status? I can't seem to find that info. > >>> The idrac on my file server doesn't seem to exhibit the same problem, > >>> and I suspect that is because it isn't being queried. > >>> > >> Hi, > > > > fence agent for IPMI is based on ipmitool. So if ping/ipmitool is not > > working there is not much to do about it. I don't know enough about oVirt > > engine but there is no real place where fence agent can memory leak because > > it does not run as daemon. > > > > m, > > > _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users