10:33 a.m.
--=-9+ieBGOxqFuvRfepr19m
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Di, 2014-09-30 at 16:04 +0100, Dan Kenigsberg wrote:
On Tue, Sep 30, 2014 at 10:17:59AM +0000, Daniel Helgenberger wrote:
>=20
> On 30.09.2014 11:57, Piotr Kliczewski wrote:
> >
> >
> >
> > ----- Original Message -----
> >> From: "Daniel Helgenberger" <daniel.helgenberger(a)m-box.de>
> >> To: "Piotr Kliczewski" <pkliczew(a)redhat.com>, "Dan
Kenigsberg" <dank=
en(a)redhat.com>
> >> Cc: "Francesco Romani"
<fromani(a)redhat.com>, users(a)ovirt.org
> >> Sent: Tuesday, September 30, 2014 11:50:28 AM
> >> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
> >>
> >> Hello Piotr,
> >>
> >> On 30.09.2014 08:37, Piotr Kliczewski wrote:
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Dan Kenigsberg" <danken(a)redhat.com>
> >>>> To: "Daniel Helgenberger"
<daniel.helgenberger(a)m-box.de>,
> >>>> pkliczew(a)redhat.com
> >>>> Cc: "Francesco Romani" <fromani(a)redhat.com>,
users(a)ovirt.org
> >>>> Sent: Tuesday, September 30, 2014 1:11:42 AM
> >>>> Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
> >>>>
> >>>> On Mon, Sep 29, 2014 at 09:02:19PM +0000, Daniel Helgenberger wrot=
e:
> >>>>> Hello Francesco,
> >>>>>
> >>>>> --
> >>>>> Daniel Helgenberger
> >>>>> m box bewegtbild GmbH
> >>>>>
> >>>>> P: +49/30/2408781-22
> >>>>> F: +49/30/2408781-10
> >>>>> ACKERSTR. 19
> >>>>> D-10115 BERLIN
> >>>>> www.m-box.de www.monkeymen.tv
> >>>>>
> >>>>>> On 29.09.2014, at 22:19, Francesco Romani
<fromani(a)redhat.com> w=
rote:
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Daniel Helgenberger"
<daniel.helgenberger(a)m-box.de>
> >>>>>>> To: "Francesco Romani"
<fromani(a)redhat.com>
> >>>>>>> Cc: "Dan Kenigsberg"
<danken(a)redhat.com>, users(a)ovirt.org
> >>>>>>> Sent: Monday, September 29, 2014 2:54:13 PM
> >>>>>>> Subject: Re: [ovirt-users] 3.4: VDSM Memory
consumption
> >>>>>>>
> >>>>>>> Hello Francesco,
> >>>>>>>
> >>>>>>>> On 29.09.2014 13:55, Francesco Romani wrote:
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> From: "Daniel Helgenberger"
<daniel.helgenberger(a)m-box.de>
> >>>>>>>>> To: "Dan Kenigsberg"
<danken(a)redhat.com>
> >>>>>>>>> Cc: users(a)ovirt.org
> >>>>>>>>> Sent: Monday, September 29, 2014 12:25:22 PM
> >>>>>>>>> Subject: Re: [ovirt-users] 3.4: VDSM Memory
consumption
> >>>>>>>>>
> >>>>>>>>> Dan,
> >>>>>>>>>
> >>>>>>>>> I just reply to the list since I do not want to
clutter BZ:
> >>>>>>>>>
> >>>>>>>>> While migrating VMs is easy (and the sampling
is already runn=
ing),
> >>>>>>>>> can
> >>>>>>>>> someone tell me the correct polling port to
block with iptabl=
es?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>> Hi Daniel,
> >>>>>>>>
> >>>>>>>> there is indeed a memory profiling patch under
discussion:
> >>>>>>>> http://gerrit.ovirt.org/#/c/32019/
> >>>>>>>>
> >>>>>>>> but for your case we'll need a backport to
3.4.x and clearer i=
nstall
> >>>>>>>> instructions,
> >>>>>>>> which I'll prepare as soon as possible.
> >>>>>>> I updated the BZ (and are now blocking 54321/tcp on one
of my h=
osts).
> >>>>>>> and verified it is not reachable.
As general info: This system=
I am
> >>>>>>> using is my LAB / Test / eval setup
for a final deployment for =
ovirt
> >>>>>>> (then 3.5) in production; so it
will go away some time in the f=
uture (a
> >>>>>>> few weeks / months). If I am the
only one experiencing this pro=
blem
> >>>>>>> then
> >>>>>>> you might be better of allocating resources elsewhere
;)
> >>>>>> Thanks for your understanding :)
> >>>>>>
> >>>>>> Unfortunately it is true that developer resources
aren't so abun=
dant,
> >>>>>> but it is also true that memleaks
should never be discarded easi=
ly and
> >>>>>> without
> >>>>>> due investigation, considering the nature and the role of
VDSM.
> >>>>>>
> >>>>>> So, I'm all in for further investigation regarding this
issue.
> >>>>>>
> >>>>>>>> As for your question: if I understood correctly
what you are a=
sking
> >>>>>>>> (still catching up the thread),
if you are trying to rule out =
the
> >>>>>>>> stats
> >>>>>>>> polling
> >>>>>>>> made by Engine to this bad leak, one simple way to
test is jus=
t to
> >>>>>>>> shutdown
> >>>>>>>> Engine,
> >>>>>>>> and let VDSMs run unguarded on hypervisors.
You'll be able to =
command
> >>>>>>>> these
> >>>>>>>> VDSMs using vdsClient or restarting Engine.
> >>>>>>> As I said in my BZ comment this is not an option right
now, but=
if
> >>>>>>> understand the matter correctly
IPTABLES reject should ultimate=
ly do
> >>>>>>> the
> >>>>>>> same?
> >>>>>> Definitely yes! Just do whatever it is more convenient for
you.
> >>>>>>
> >>>>> As you might have already seen in the BZ comment the leak
stopped=
after
> >>>>> blocking the port. Though this is clearly
no permanent option - p=
lease
> >>>>> let
> >>>>> me know if I can be of any more assistance!
> >>>> The immediate suspect in this situation is M2Crypto. Could you ver=
ify
> >>>> that by re-opening the firewall and setting
ssl=3DFalse in vdsm.co=
nf?
> >>>>
> >>>> You should disable ssl on Engine side and restart both Engine and =
Vdsm
> >>>> (too bad I do not recall how that's done on
Engine: Piotr, can you=
help?).
> >>>>
> >>> In vdc_options table there is option EncryptHostCommunication.
> >> Please confirm the following procedure is correct:
> >>
> >> 1. Change Postgres table value:
> >> # sudo -u postgres psql -U postgres engine -c "update vdc_options se=
t
> >> option_value =3D 'false' where option_name =3D
'EncryptHostCommunica=
tion';"
> >> engine=3D# SELECT * from vdc_options where
> >> option_name=3D'EncryptHostCommunication';
> >> option_id | option_name | option_value | version
> >> -----------+--------------------------+--------------+---------
> >> 335 | EncryptHostCommunication | false | general
> >> (1 row)
> >>
> >> 2. Restart engine
> >> 3. On the hosts;
> >> grep ssl /etc/vdsm/vdsm.conf
> >> #ssl =3D true
> >> ssl =3D false
> >>
> >> 4. restart VDSM
> >>
> >> I assume I have to set 'ssl =3D false' this on on all hosts?
> >>> Please to set it to false and restart the engine.
> >>>
> > I believe that you need to update a bit more on vdsm side.
> Indeed your beliefs were true; I was already running in this error whil=
e
> starting vdsm:
> vdsm: Running validate_configuration
> FAILED: conflicting vdsm and libvirt-qemu tls configuration.
> vdsm.conf with ssl=3DFalse requires the following changed:
> libvirtd.conf: listen_tcp=3D1, auth_tcp=3D"none",
> qemu.conf: spice_tls=3D0.
>=20
> After changing the apropriate lines vdsm is running again.
>=20
> @Dan: As soon as I have more info I will update BZ, and if this test
> comes up negatve I would reverse the changes?
> I suppose 'Reinstalling' the host will revert this?
=20
Yes - after you revert the config change on Engine's side.
=20
(thanks for your help, /me is looking forward to blaming ssl)
Glad I can help you
Dan, literally in every aspect ;)
I am updating BZ in a few minututs with the details, but so far no RSS
growth after disabling SSL.
=20
--=20
Daniel Helgenberger=20
m box bewegtbild GmbH=20
P: +49/30/2408781-22
F: +49/30/2408781-10
ACKERSTR. 19=20
D-10115 BERLIN=20
www.m-box.de www.monkeymen.tv=20
Gesch=C3=A4ftsf=C3=BChrer: Martin Retschitzegger / Michaela G=C3=B6llner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767=20
--=-9+ieBGOxqFuvRfepr19m
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINtjCCBBYw
ggL+oAMCAQICCwQAAAAAAS9O4S9SMA0GCSqGSIb3DQEBBQUAMFcxCzAJBgNVBAYTAkJFMRkwFwYD
VQQKExBHbG9iYWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT
aWduIFJvb3QgQ0EwHhcNMTEwNDEzMTAwMDAwWhcNMTkwNDEzMTAwMDAwWjBUMQswCQYDVQQGEwJC
RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQZXJzb25h
bFNpZ24gMiBDQSAtIEcyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwWtB+TXs+BJ9
3SJRaV+3uRNGJ3cUO+MTgW8+5HQXfgy19CzkDI1T1NwwICi/bo4R/mYR5FEWx91//eE0ElC/89iY
7GkL0tDasmVx4TOXnrqrsziUcxEPPqHRE8x4NhtBK7+8o0nsMIJMA1gyZ2FA5To2Ew1BBuvovvDJ
+Nua3qOCNBNu+8A+eNpJlVnlu/qB7+XWaPXtUMlsIikxD+gREFVUgYE4VzBuLa2kkg0VLd09XkE2
ceRDm6YgRATuDk6ogUyX4OLxCGIJF8yi6Z37M0wemDA6Uff0EuqdwDQd5HwG/rernUjt1grLdAxq
8BwywRRg0eFHmE+ShhpyO3Fi+wIDAQABo4HlMIHiMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8E
CDAGAQH/AgEAMB0GA1UdDgQWBBQ/FdJtfC/nMZ5DCgaolGwsO8XuZTBHBgNVHSAEQDA+MDwGBFUd
IAAwNDAyBggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8w
MwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5nbG9iYWxzaWduLm5ldC9yb290LmNybDAfBgNV
HSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0BAQUFAAOCAQEAQ3N5zKTMSTED
HGFAgd/gu91Kb8AxPHgjq+7dhf7mkCinMqqrLai2XOrz8CP63BPaAx7oGOUBI0MyASBGk5zej9L3
oHtiF2BL01m1sBnT8rQxT2CJd/+jqjUl0p2ew8T3HSyatrsooGvDwf00yCB2JHTNvtQxNO8t6x/+
048A1Q+0i7uf0nTnyrJLjD04zhL89ytetZspltOpJVYbmwiFjq6PxsdUNthUDme/9pOLmKDnQU0p
W/JEwLs2TYCBNKwdgSGAk8/z+s2SCltKIG0Uh5U6t6j7JPuwNP/znImwMrlHDJ1YpW0rkF2PGraV
CgDBf9dOB+IIpnwHfIi+LD+eITCCBMowggOyoAMCAQICEQCWaWbA3qWpL+Qmn6I16DynMA0GCSqG
SIb3DQEBBQUAMFQxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYD
VQQDEyFHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gRzIwHhcNMTMwODI3MTY1NzU4WhcN
MTYwODI3MTY1NzU4WjBYMQswCQYDVQQGEwJERTEcMBoGA1UEAxMTRGFuaWVsIEhlbGdlbmJlcmdl
cjErMCkGCSqGSIb3DQEJARYcZGFuaWVsLmhlbGdlbmJlcmdlckBtLWJveC5kZTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAM4BQ5vPknk1OGLd1qKSUIKmQLrjccjJcYj7qtAtA+fNYKF8
9p1VY4UwiFcF9jKlmA9Q8o8tYSx16LYYFoGWokNRAeKFXZiBZiHyI0ekpEfxo8N5cTMCcxKcSYWV
8sqzmBPCoMNpmiVoC8ec8Nv5SqXH34VVtDmNLfiVlsTyomBXAJkJ2/n5XqJzPLFGWWREtPLkVVS+
u426vt/hNsQi5akNoidYeXo98JcrmeApFJ3zB2KxvMziHx8LD4q1gAl9NumtX5YLbCpdWL9AkWdX
Oaro3D9zj6Q6LyGwa/UQUrZdg3BXc07hjHZn6d9vet1SzpbyqQpTzM63yXiX1meEMlMCAwEAAaOC
AZEwggGNMA4GA1UdDwEB/wQEAwIFoDBMBgNVHSAERTBDMEEGCSsGAQQBoDIBKDA0MDIGCCsGAQUF
BwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAnBgNVHREEIDAegRxk
YW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94LmRlMAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUH
AwIGCCsGAQUFBwMEMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20v
Z3MvZ3NwZXJzb25hbHNpZ24yZzIuY3JsMFUGCCsGAQUFBwEBBEkwRzBFBggrBgEFBQcwAoY5aHR0
cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NwZXJzb25hbHNpZ24yZzIuY3J0MB0G
A1UdDgQWBBS8NFA/upd+Wipw2nj8RD/Ct+R2GTAfBgNVHSMEGDAWgBQ/FdJtfC/nMZ5DCgaolGws
O8XuZTANBgkqhkiG9w0BAQUFAAOCAQEAXVTpu4fhOLETAW0zdbQiIwBIMZgeVNJnWV3GsMxByycU
63P+WBQTBl9qj47vHLmVdeF7MzH0QSXZSc9Tnfr6CYIImpyIZxRAGpAsWmtZf3JieRA0+j4GQJF2
zAea1NXYXoG9+ZSSZHBSxKUdrRdVdE320nuVGTT2HjEI2LEYbOvaXyi6HhpuHUiyu4LD0+RIT3fi
T8jUiKKLTsApTD+Ak8SLF0IESOSA6htirv69mDDC7Klg9dT7QBPO7dpoKIUOldV3VhahndVfsDff
KD7pkUUvG5XftYEQOxlWDJzuTBeqf/4hxXMtzFU9OaI6oKJjLfr6B+XBc6xwOtc/NMWmejCCBMow
ggOyoAMCAQICEQCWaWbA3qWpL+Qmn6I16DynMA0GCSqGSIb3DQEBBQUAMFQxCzAJBgNVBAYTAkJF
MRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYDVQQDEyFHbG9iYWxTaWduIFBlcnNvbmFs
U2lnbiAyIENBIC0gRzIwHhcNMTMwODI3MTY1NzU4WhcNMTYwODI3MTY1NzU4WjBYMQswCQYDVQQG
EwJERTEcMBoGA1UEAxMTRGFuaWVsIEhlbGdlbmJlcmdlcjErMCkGCSqGSIb3DQEJARYcZGFuaWVs
LmhlbGdlbmJlcmdlckBtLWJveC5kZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM4B
Q5vPknk1OGLd1qKSUIKmQLrjccjJcYj7qtAtA+fNYKF89p1VY4UwiFcF9jKlmA9Q8o8tYSx16LYY
FoGWokNRAeKFXZiBZiHyI0ekpEfxo8N5cTMCcxKcSYWV8sqzmBPCoMNpmiVoC8ec8Nv5SqXH34VV
tDmNLfiVlsTyomBXAJkJ2/n5XqJzPLFGWWREtPLkVVS+u426vt/hNsQi5akNoidYeXo98JcrmeAp
FJ3zB2KxvMziHx8LD4q1gAl9NumtX5YLbCpdWL9AkWdXOaro3D9zj6Q6LyGwa/UQUrZdg3BXc07h
jHZn6d9vet1SzpbyqQpTzM63yXiX1meEMlMCAwEAAaOCAZEwggGNMA4GA1UdDwEB/wQEAwIFoDBM
BgNVHSAERTBDMEEGCSsGAQQBoDIBKDA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxz
aWduLmNvbS9yZXBvc2l0b3J5LzAnBgNVHREEIDAegRxkYW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94
LmRlMAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMEMGA1UdHwQ8MDow
OKA2oDSGMmh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vZ3MvZ3NwZXJzb25hbHNpZ24yZzIuY3Js
MFUGCCsGAQUFBwEBBEkwRzBFBggrBgEFBQcwAoY5aHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNv
bS9jYWNlcnQvZ3NwZXJzb25hbHNpZ24yZzIuY3J0MB0GA1UdDgQWBBS8NFA/upd+Wipw2nj8RD/C
t+R2GTAfBgNVHSMEGDAWgBQ/FdJtfC/nMZ5DCgaolGwsO8XuZTANBgkqhkiG9w0BAQUFAAOCAQEA
XVTpu4fhOLETAW0zdbQiIwBIMZgeVNJnWV3GsMxByycU63P+WBQTBl9qj47vHLmVdeF7MzH0QSXZ
Sc9Tnfr6CYIImpyIZxRAGpAsWmtZf3JieRA0+j4GQJF2zAea1NXYXoG9+ZSSZHBSxKUdrRdVdE32
0nuVGTT2HjEI2LEYbOvaXyi6HhpuHUiyu4LD0+RIT3fiT8jUiKKLTsApTD+Ak8SLF0IESOSA6hti
rv69mDDC7Klg9dT7QBPO7dpoKIUOldV3VhahndVfsDffKD7pkUUvG5XftYEQOxlWDJzuTBeqf/4h
xXMtzFU9OaI6oKJjLfr6B+XBc6xwOtc/NMWmejGCAucwggLjAgEBMGkwVDELMAkGA1UEBhMCQkUx
GTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24gUGVyc29uYWxT
aWduIDIgQ0EgLSBHMgIRAJZpZsDepakv5CafojXoPKcwCQYFKw4DAhoFAKCCAVMwGAYJKoZIhvcN
AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwOTMwMTUzMzQxWjAjBgkqhkiG9w0B
CQQxFgQUYd8YsPl/Yj5jwbK40CsZf8ch7DkweAYJKwYBBAGCNxAEMWswaTBUMQswCQYDVQQGEwJC
RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQZXJzb25h
bFNpZ24gMiBDQSAtIEcyAhEAlmlmwN6lqS/kJp+iNeg8pzB6BgsqhkiG9w0BCRACCzFroGkwVDEL
MAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNp
Z24gUGVyc29uYWxTaWduIDIgQ0EgLSBHMgIRAJZpZsDepakv5CafojXoPKcwDQYJKoZIhvcNAQEB
BQAEggEADhFXp3I2NnYdCtmgtRH+N9tsKMvKjUFK8vrezXinaMHvOjQwaiyWeVzyvlM4XMhvDrtK
kGx4K+LfgsWCNLWEjZwLAT2nCy8yRv+5NwhEKFTOcxm84gG+5PZrIa44TMP3l9K7vAXRkjQkZU1+
gmdBGQKxlXbilVPNbO7vYV4wDzJkwnqp/rp25iu6mgT3J1raMwE94BTnJ3EkfEzB+IMyhq7Je/II
Re2emjWIUavUpfaYdpaF8I7CvBHz/fgfTa62/ImAgwQF/iVar+JjJkvvFXEE39qBw2XaWG4tqUCv
WicjPB6fwNZuPu+1JFt/VCE/8uGUWUu5/X4qoEtXd1TmUQAAAAAAAA==
--=-9+ieBGOxqFuvRfepr19m--