[ovirt-users] ?3.4: VDSM Memory consumption

Tue Sep 30 10:23:47 UTC 2014

On 30.09.2014 11:57, Piotr Kliczewski wrote:
>
>
>
> ----- Original Message -----
>> From: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>
>> To: "Piotr Kliczewski" <pkliczew at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>
>> Cc: "Francesco Romani" <fromani at redhat.com>, users at ovirt.org
>> Sent: Tuesday, September 30, 2014 11:50:28 AM
>> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
>>
>> Hello Piotr,
>>
>> On 30.09.2014 08:37, Piotr Kliczewski wrote:
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Dan Kenigsberg" <danken at redhat.com>
>>>> To: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>,
>>>> pkliczew at redhat.com
>>>> Cc: "Francesco Romani" <fromani at redhat.com>, users at ovirt.org
>>>> Sent: Tuesday, September 30, 2014 1:11:42 AM
>>>> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
>>>>
>>>> On Mon, Sep 29, 2014 at 09:02:19PM +0000, Daniel Helgenberger wrote:
>>>>> Hello Francesco,
>>>>>
>>>>> --
>>>>> Daniel Helgenberger
>>>>> m box bewegtbild GmbH
>>>>>
>>>>> P: +49/30/2408781-22
>>>>> F: +49/30/2408781-10
>>>>> ACKERSTR. 19
>>>>> D-10115 BERLIN
>>>>> www.m-box.de  www.monkeymen.tv
>>>>>
>>>>>> On 29.09.2014, at 22:19, Francesco Romani <fromani at redhat.com> wrote:
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>
>>>>>>> To: "Francesco Romani" <fromani at redhat.com>
>>>>>>> Cc: "Dan Kenigsberg" <danken at redhat.com>, users at ovirt.org
>>>>>>> Sent: Monday, September 29, 2014 2:54:13 PM
>>>>>>> Subject: Re: [ovirt-users]    3.4: VDSM Memory consumption
>>>>>>>
>>>>>>> Hello Francesco,
>>>>>>>
>>>>>>>> On 29.09.2014 13:55, Francesco Romani wrote:
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>
>>>>>>>>> To: "Dan Kenigsberg" <danken at redhat.com>
>>>>>>>>> Cc: users at ovirt.org
>>>>>>>>> Sent: Monday, September 29, 2014 12:25:22 PM
>>>>>>>>> Subject: Re: [ovirt-users]    3.4: VDSM Memory consumption
>>>>>>>>>
>>>>>>>>> Dan,
>>>>>>>>>
>>>>>>>>> I just reply to the list since I do not want to clutter BZ:
>>>>>>>>>
>>>>>>>>> While migrating VMs is easy (and the sampling is already running),
>>>>>>>>> can
>>>>>>>>> someone tell me the correct polling port to block with iptables?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>> Hi Daniel,
>>>>>>>>
>>>>>>>> there is indeed a memory profiling patch under discussion:
>>>>>>>> http://gerrit.ovirt.org/#/c/32019/
>>>>>>>>
>>>>>>>> but for your case we'll need a backport to 3.4.x and clearer install
>>>>>>>> instructions,
>>>>>>>> which I'll prepare as soon as possible.
>>>>>>> I updated the BZ (and are now blocking 54321/tcp on one of my hosts).
>>>>>>> and verified it is not reachable. As  general info: This system I am
>>>>>>> using is my LAB / Test / eval setup for a final deployment for ovirt
>>>>>>> (then 3.5) in production; so it will go away some time in the future (a
>>>>>>> few weeks / months). If I am the only one experiencing this problem
>>>>>>> then
>>>>>>> you might be better of allocating resources elsewhere ;)
>>>>>> Thanks for your understanding :)
>>>>>>
>>>>>> Unfortunately it is true that developer resources aren't so abundant,
>>>>>> but it is also true that memleaks should never be discarded easily and
>>>>>> without
>>>>>> due investigation, considering the nature and the role of VDSM.
>>>>>>
>>>>>> So, I'm all in for further investigation regarding this issue.
>>>>>>
>>>>>>>> As for your question: if I understood correctly what you are asking
>>>>>>>> (still catching up the thread), if you are trying to rule out the
>>>>>>>> stats
>>>>>>>> polling
>>>>>>>> made by Engine to this bad leak, one simple way to test is just to
>>>>>>>> shutdown
>>>>>>>> Engine,
>>>>>>>> and let VDSMs run unguarded on hypervisors. You'll be able to command
>>>>>>>> these
>>>>>>>> VDSMs using vdsClient or restarting Engine.
>>>>>>> As I said in my BZ comment this is not an option right now, but if
>>>>>>> understand the matter correctly IPTABLES reject should ultimately do
>>>>>>> the
>>>>>>> same?
>>>>>> Definitely yes! Just do whatever it is more convenient for you.
>>>>>>
>>>>> As you might have already seen in the BZ comment the leak stopped after
>>>>> blocking the port. Though this is clearly no permanent option - please
>>>>> let
>>>>> me know if I can be of any more assistance!
>>>> The immediate suspect in this situation is M2Crypto. Could you verify
>>>> that by re-opening the firewall and setting ssl=False in vdsm.conf?
>>>>
>>>> You should disable ssl on Engine side and restart both Engine and Vdsm
>>>> (too bad I do not recall how that's done on Engine: Piotr, can you help?).
>>>>
>>> In vdc_options table there is option EncryptHostCommunication.
>> Please confirm the following procedure is correct:
>>
>> 1. Change Postgres table value:
>> # sudo -u postgres psql -U postgres engine -c "update vdc_options set
>> option_value = 'false' where option_name = 'EncryptHostCommunication';"
>> engine=# SELECT * from vdc_options where
>> option_name='EncryptHostCommunication';
>>  option_id |       option_name        | option_value | version
>> -----------+--------------------------+--------------+---------
>>        335 | EncryptHostCommunication | false        | general
>> (1 row)
>>
>> 2. Restart engine
>> 3. On the hosts;
>> grep ssl /etc/vdsm/vdsm.conf
>> #ssl = true
>> ssl = false
>>
>> 4. restart VDSM
>>
>> I assume I have to set 'ssl = false' this on on all hosts?
>>> Please to set it to false and restart the engine.
>>>
> I believe that you need to update a bit more on vdsm side.
> Please follow [1] section "Configure ovirt-engine and vdsm to work in non-secure mode"
>
> There is wrong name of the option and it should be EncryptHostCommunication.
>
> [1] http://www.ovirt.org/Developers_All_In_One
I forgot; I suppose hosted-engine-ha is out of order because of disabled
ssl?
hosted-engine --connect-storage
Connecting Storage Server
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 2578, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 712, in connectStorageServer
    res = self.s.connectStorageServer(serverType, spUUID, conList)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py",
line 195, in connect
    cert_reqs=self.cert_reqs)
  File "/usr/lib64/python2.6/ssl.py", line 342, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs)
  File "/usr/lib64/python2.6/ssl.py", line 120, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.6/ssl.py", line 279, in do_handshake
    self._sslobj.do_handshake()
SSLError: [Errno 8] _ssl.c:492: EOF occurred in violation of protocol

>
>> --
>> Daniel Helgenberger
>> m box bewegtbild GmbH
>>
>> P: +49/30/2408781-22
>> F: +49/30/2408781-10
>>
>> ACKERSTR. 19
>> D-10115 BERLIN
>>
>>
>> www.m-box.de  www.monkeymen.tv
>>
>> Geschäftsführer: Martin Retschitzegger / Michaela Göllner
>> Handeslregister: Amtsgericht Charlottenburg / HRB 112767
>>
>>

-- 
Daniel Helgenberger
m box bewegtbild GmbH

P: +49/30/2408781-22
F: +49/30/2408781-10

ACKERSTR. 19
D-10115 BERLIN


www.m-box.de  www.monkeymen.tv

Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767