Re: [ovirt-users] ?3.4: VDSM Memory consumption

On Tue, Sep 30, 2014 at 10:23:47AM +0000, Daniel Helgenberger wrote: > > On 30.09.2014 11:57, Piotr Kliczewski wrote: >> >> >> >> ----- Original Message ----- >>> From: "Daniel Helgenberger" <daniel.helgenberger(a)m-box.de&gt; >>> To: "Piotr Kliczewski" <pkliczew(a)redhat.com&gt;, "Dan Kenigsberg" <danken(a)redhat.com&gt; >>> Cc: "Francesco Romani" <fromani(a)redhat.com&gt;, users(a)ovirt.org >>> Sent: Tuesday, September 30, 2014 11:50:28 AM >>> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption >>> >>> Hello Piotr, >>> >>> On 30.09.2014 08:37, Piotr Kliczewski wrote: >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Dan Kenigsberg" <danken(a)redhat.com&gt; >>>>> To: "Daniel Helgenberger" <daniel.helgenberger(a)m-box.de&gt;, >>>>> pkliczew(a)redhat.com >>>>> Cc: "Francesco Romani" <fromani(a)redhat.com&gt;, users(a)ovirt.org >>>>> Sent: Tuesday, September 30, 2014 1:11:42 AM >>>>> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption >>>>> >>>>> On Mon, Sep 29, 2014 at 09:02:19PM +0000, Daniel Helgenberger wrote: >>>>>> Hello Francesco, >>>>>> >>>>>> -- >>>>>> Daniel Helgenberger >>>>>> m box bewegtbild GmbH >>>>>> >>>>>> P: +49/30/2408781-22 >>>>>> F: +49/30/2408781-10 >>>>>> ACKERSTR. 19 >>>>>> D-10115 BERLIN >>>>>> www.m-box.de www.monkeymen.tv >>>>>> >>>>>>> On 29.09.2014, at 22:19, Francesco Romani <fromani(a)redhat.com&gt; wrote: >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Daniel Helgenberger" <daniel.helgenberger(a)m-box.de&gt; >>>>>>>> To: "Francesco Romani" <fromani(a)redhat.com&gt; >>>>>>>> Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, users(a)ovirt.org >>>>>>>> Sent: Monday, September 29, 2014 2:54:13 PM >>>>>>>> Subject: Re: [ovirt-users] 3.4: VDSM Memory consumption >>>>>>>> >>>>>>>> Hello Francesco, >>>>>>>> >>>>>>>>> On 29.09.2014 13:55, Francesco Romani wrote: >>>>>>>>> ----- Original Message ----- >>>>>>>>>> From: "Daniel Helgenberger" <daniel.helgenberger(a)m-box.de&gt; >>>>>>>>>> To: "Dan Kenigsberg" <danken(a)redhat.com&gt; >>>>>>>>>> Cc: users(a)ovirt.org >>>>>>>>>> Sent: Monday, September 29, 2014 12:25:22 PM >>>>>>>>>> Subject: Re: [ovirt-users] 3.4: VDSM Memory consumption >>>>>>>>>> >>>>>>>>>> Dan, >>>>>>>>>> >>>>>>>>>> I just reply to the list since I do not want to clutter BZ: >>>>>>>>>> >>>>>>>>>> While migrating VMs is easy (and the sampling is already running), >>>>>>>>>> can >>>>>>>>>> someone tell me the correct polling port to block with iptables? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>> Hi Daniel, >>>>>>>>> >>>>>>>>> there is indeed a memory profiling patch under discussion: >>>>>>>>> http://gerrit.ovirt.org/#/c/32019/ >>>>>>>>> >>>>>>>>> but for your case we'll need a backport to 3.4.x and clearer install >>>>>>>>> instructions, >>>>>>>>> which I'll prepare as soon as possible. >>>>>>>> I updated the BZ (and are now blocking 54321/tcp on one of my hosts). >>>>>>>> and verified it is not reachable. As general info: This system I am >>>>>>>> using is my LAB / Test / eval setup for a final deployment for ovirt >>>>>>>> (then 3.5) in production; so it will go away some time in the future (a >>>>>>>> few weeks / months). If I am the only one experiencing this problem >>>>>>>> then >>>>>>>> you might be better of allocating resources elsewhere ;) >>>>>>> Thanks for your understanding :) >>>>>>> >>>>>>> Unfortunately it is true that developer resources aren't so abundant, >>>>>>> but it is also true that memleaks should never be discarded easily and >>>>>>> without >>>>>>> due investigation, considering the nature and the role of VDSM. >>>>>>> >>>>>>> So, I'm all in for further investigation regarding this issue. >>>>>>> >>>>>>>>> As for your question: if I understood correctly what you are asking >>>>>>>>> (still catching up the thread), if you are trying to rule out the >>>>>>>>> stats >>>>>>>>> polling >>>>>>>>> made by Engine to this bad leak, one simple way to test is just to >>>>>>>>> shutdown >>>>>>>>> Engine, >>>>>>>>> and let VDSMs run unguarded on hypervisors. You'll be able to command >>>>>>>>> these >>>>>>>>> VDSMs using vdsClient or restarting Engine. >>>>>>>> As I said in my BZ comment this is not an option right now, but if >>>>>>>> understand the matter correctly IPTABLES reject should ultimately do >>>>>>>> the >>>>>>>> same? >>>>>>> Definitely yes! Just do whatever it is more convenient for you. >>>>>>> >>>>>> As you might have already seen in the BZ comment the leak stopped after >>>>>> blocking the port. Though this is clearly no permanent option - please >>>>>> let >>>>>> me know if I can be of any more assistance! >>>>> The immediate suspect in this situation is M2Crypto. Could you verify >>>>> that by re-opening the firewall and setting ssl=False in vdsm.conf? >>>>> >>>>> You should disable ssl on Engine side and restart both Engine and Vdsm >>>>> (too bad I do not recall how that's done on Engine: Piotr, can you help?). >>>>> >>>> In vdc_options table there is option EncryptHostCommunication. >>> Please confirm the following procedure is correct: >>> >>> 1. Change Postgres table value: >>> # sudo -u postgres psql -U postgres engine -c "update vdc_options set >>> option_value = 'false' where option_name = 'EncryptHostCommunication';" >>> engine=# SELECT * from vdc_options where >>> option_name='EncryptHostCommunication'; >>> option_id | option_name | option_value | version >>> -----------+--------------------------+--------------+--------- >>> 335 | EncryptHostCommunication | false | general >>> (1 row) >>> >>> 2. Restart engine >>> 3. On the hosts; >>> grep ssl /etc/vdsm/vdsm.conf >>> #ssl = true >>> ssl = false >>> >>> 4. restart VDSM >>> >>> I assume I have to set 'ssl = false' this on on all hosts? >>>> Please to set it to false and restart the engine. >>>> >> I believe that you need to update a bit more on vdsm side. >> Please follow [1] section "Configure ovirt-engine and vdsm to work in non-secure mode" >> >> There is wrong name of the option and it should be EncryptHostCommunication. >> >> [1] http://www.ovirt.org/Developers_All_In_One > I forgot; I suppose hosted-engine-ha is out of order because of disabled > ssl? Indeed. And in hosted-engine, too, I need someone else's help (Sando?) to tell how to disable ssl.

> hosted-engine --connect-storage > Connecting Storage Server > Traceback (most recent call last): > File "/usr/share/vdsm/vdsClient.py", line 2578, in <module> > code, message = commands[command][0](commandArgs) > File "/usr/share/vdsm/vdsClient.py", line 712, in connectStorageServer > res = self.s.connectStorageServer(serverType, spUUID, conList) > File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ > return self.__send(self.__name, args) > File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request > verbose=self.__verbose > File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request > self.send_content(h, request_body) > File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content > connection.endheaders() > File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders > self._send_output() > File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output > self.send(msg) > File "/usr/lib64/python2.6/httplib.py", line 739, in send > self.connect() > File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", > line 195, in connect > cert_reqs=self.cert_reqs) > File "/usr/lib64/python2.6/ssl.py", line 342, in wrap_socket > suppress_ragged_eofs=suppress_ragged_eofs) > File "/usr/lib64/python2.6/ssl.py", line 120, in __init__ > self.do_handshake() > File "/usr/lib64/python2.6/ssl.py", line 279, in do_handshake > self._sslobj.do_handshake() > SSLError: [Errno 8] _ssl.c:492: EOF occurred in violation of protocol