[ovirt-users] ?3.4: VDSM Memory consumption

Tue Sep 30 09:57:25 UTC 2014

----- Original Message -----
> From: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>
> To: "Piotr Kliczewski" <pkliczew at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>
> Cc: "Francesco Romani" <fromani at redhat.com>, users at ovirt.org
> Sent: Tuesday, September 30, 2014 11:50:28 AM
> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
> 
> Hello Piotr,
> 
> On 30.09.2014 08:37, Piotr Kliczewski wrote:
> >
> >
> >
> > ----- Original Message -----
> >> From: "Dan Kenigsberg" <danken at redhat.com>
> >> To: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>,
> >> pkliczew at redhat.com
> >> Cc: "Francesco Romani" <fromani at redhat.com>, users at ovirt.org
> >> Sent: Tuesday, September 30, 2014 1:11:42 AM
> >> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
> >>
> >> On Mon, Sep 29, 2014 at 09:02:19PM +0000, Daniel Helgenberger wrote:
> >>> Hello Francesco,
> >>>
> >>> --
> >>> Daniel Helgenberger
> >>> m box bewegtbild GmbH
> >>>
> >>> P: +49/30/2408781-22
> >>> F: +49/30/2408781-10
> >>> ACKERSTR. 19
> >>> D-10115 BERLIN
> >>> www.m-box.de  www.monkeymen.tv
> >>>
> >>>> On 29.09.2014, at 22:19, Francesco Romani <fromani at redhat.com> wrote:
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>
> >>>>> To: "Francesco Romani" <fromani at redhat.com>
> >>>>> Cc: "Dan Kenigsberg" <danken at redhat.com>, users at ovirt.org
> >>>>> Sent: Monday, September 29, 2014 2:54:13 PM
> >>>>> Subject: Re: [ovirt-users]    3.4: VDSM Memory consumption
> >>>>>
> >>>>> Hello Francesco,
> >>>>>
> >>>>>> On 29.09.2014 13:55, Francesco Romani wrote:
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Daniel Helgenberger" <daniel.helgenberger at m-box.de>
> >>>>>>> To: "Dan Kenigsberg" <danken at redhat.com>
> >>>>>>> Cc: users at ovirt.org
> >>>>>>> Sent: Monday, September 29, 2014 12:25:22 PM
> >>>>>>> Subject: Re: [ovirt-users]    3.4: VDSM Memory consumption
> >>>>>>>
> >>>>>>> Dan,
> >>>>>>>
> >>>>>>> I just reply to the list since I do not want to clutter BZ:
> >>>>>>>
> >>>>>>> While migrating VMs is easy (and the sampling is already running),
> >>>>>>> can
> >>>>>>> someone tell me the correct polling port to block with iptables?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>> Hi Daniel,
> >>>>>>
> >>>>>> there is indeed a memory profiling patch under discussion:
> >>>>>> http://gerrit.ovirt.org/#/c/32019/
> >>>>>>
> >>>>>> but for your case we'll need a backport to 3.4.x and clearer install
> >>>>>> instructions,
> >>>>>> which I'll prepare as soon as possible.
> >>>>> I updated the BZ (and are now blocking 54321/tcp on one of my hosts).
> >>>>> and verified it is not reachable. As  general info: This system I am
> >>>>> using is my LAB / Test / eval setup for a final deployment for ovirt
> >>>>> (then 3.5) in production; so it will go away some time in the future (a
> >>>>> few weeks / months). If I am the only one experiencing this problem
> >>>>> then
> >>>>> you might be better of allocating resources elsewhere ;)
> >>>> Thanks for your understanding :)
> >>>>
> >>>> Unfortunately it is true that developer resources aren't so abundant,
> >>>> but it is also true that memleaks should never be discarded easily and
> >>>> without
> >>>> due investigation, considering the nature and the role of VDSM.
> >>>>
> >>>> So, I'm all in for further investigation regarding this issue.
> >>>>
> >>>>>> As for your question: if I understood correctly what you are asking
> >>>>>> (still catching up the thread), if you are trying to rule out the
> >>>>>> stats
> >>>>>> polling
> >>>>>> made by Engine to this bad leak, one simple way to test is just to
> >>>>>> shutdown
> >>>>>> Engine,
> >>>>>> and let VDSMs run unguarded on hypervisors. You'll be able to command
> >>>>>> these
> >>>>>> VDSMs using vdsClient or restarting Engine.
> >>>>> As I said in my BZ comment this is not an option right now, but if
> >>>>> understand the matter correctly IPTABLES reject should ultimately do
> >>>>> the
> >>>>> same?
> >>>> Definitely yes! Just do whatever it is more convenient for you.
> >>>>
> >>> As you might have already seen in the BZ comment the leak stopped after
> >>> blocking the port. Though this is clearly no permanent option - please
> >>> let
> >>> me know if I can be of any more assistance!
> >> The immediate suspect in this situation is M2Crypto. Could you verify
> >> that by re-opening the firewall and setting ssl=False in vdsm.conf?
> >>
> >> You should disable ssl on Engine side and restart both Engine and Vdsm
> >> (too bad I do not recall how that's done on Engine: Piotr, can you help?).
> >>
> > In vdc_options table there is option EncryptHostCommunication.
> Please confirm the following procedure is correct:
> 
> 1. Change Postgres table value:
> # sudo -u postgres psql -U postgres engine -c "update vdc_options set
> option_value = 'false' where option_name = 'EncryptHostCommunication';"
> engine=# SELECT * from vdc_options where
> option_name='EncryptHostCommunication';
>  option_id |       option_name        | option_value | version
> -----------+--------------------------+--------------+---------
>        335 | EncryptHostCommunication | false        | general
> (1 row)
> 
> 2. Restart engine
> 3. On the hosts;
> grep ssl /etc/vdsm/vdsm.conf
> #ssl = true
> ssl = false
> 
> 4. restart VDSM
> 
> I assume I have to set 'ssl = false' this on on all hosts?
> > Please to set it to false and restart the engine.
> >

I believe that you need to update a bit more on vdsm side.
Please follow [1] section "Configure ovirt-engine and vdsm to work in non-secure mode"

There is wrong name of the option and it should be EncryptHostCommunication.

[1] http://www.ovirt.org/Developers_All_In_One

> 
> --
> Daniel Helgenberger
> m box bewegtbild GmbH
> 
> P: +49/30/2408781-22
> F: +49/30/2408781-10
> 
> ACKERSTR. 19
> D-10115 BERLIN
> 
> 
> www.m-box.de  www.monkeymen.tv
> 
> Geschäftsführer: Martin Retschitzegger / Michaela Göllner
> Handeslregister: Amtsgericht Charlottenburg / HRB 112767
> 
> 