Il 30/09/2014 17:03, Dan Kenigsberg ha scritto:
On Tue, Sep 30, 2014 at 10:23:47AM +0000, Daniel Helgenberger wrote:
On 30.09.2014 11:57, Piotr Kliczewski wrote:
----- Original Message -----
From: "Daniel Helgenberger" <daniel.helgenberger@m-box.de> To: "Piotr Kliczewski" <pkliczew@redhat.com>, "Dan Kenigsberg" <dank=
en@redhat.com>
Cc: "Francesco Romani" <fromani@redhat.com>, users@ovirt.org Sent: Tuesday, September 30, 2014 11:50:28 AM Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption
Hello Piotr,
On 30.09.2014 08:37, Piotr Kliczewski wrote:
----- Original Message ----- > From: "Dan Kenigsberg" <danken@redhat.com> > To: "Daniel Helgenberger" <daniel.helgenberger@m-box.de>, > pkliczew@redhat.com > Cc: "Francesco Romani" <fromani@redhat.com>, users@ovirt.org > Sent: Tuesday, September 30, 2014 1:11:42 AM > Subject: Re: [ovirt-users]?3.4: VDSM Memory consumption > > On Mon, Sep 29, 2014 at 09:02:19PM +0000, Daniel Helgenberger wrot=
e:
>> Hello Francesco, >> >> -- >> Daniel Helgenberger >> m box bewegtbild GmbH >> >> P: +49/30/2408781-22 >> F: +49/30/2408781-10 >> ACKERSTR. 19 >> D-10115 BERLIN >> www.m-box.de www.monkeymen.tv >> >>> On 29.09.2014, at 22:19, Francesco Romani <fromani@redhat.com> w= rote: >>> >>> ----- Original Message ----- >>>> From: "Daniel Helgenberger" <daniel.helgenberger@m-box.de> >>>> To: "Francesco Romani" <fromani@redhat.com> >>>> Cc: "Dan Kenigsberg" <danken@redhat.com>, users@ovirt.org >>>> Sent: Monday, September 29, 2014 2:54:13 PM >>>> Subject: Re: [ovirt-users] 3.4: VDSM Memory consumption >>>> >>>> Hello Francesco, >>>> >>>>> On 29.09.2014 13:55, Francesco Romani wrote: >>>>> ----- Original Message ----- >>>>>> From: "Daniel Helgenberger" <daniel.helgenberger@m-box.de> >>>>>> To: "Dan Kenigsberg" <danken@redhat.com> >>>>>> Cc: users@ovirt.org >>>>>> Sent: Monday, September 29, 2014 12:25:22 PM >>>>>> Subject: Re: [ovirt-users] 3.4: VDSM Memory consumption >>>>>> >>>>>> Dan, >>>>>> >>>>>> I just reply to the list since I do not want to clutter BZ: >>>>>> >>>>>> While migrating VMs is easy (and the sampling is already runn= ing), >>>>>> can >>>>>> someone tell me the correct polling port to block with iptabl= es? >>>>>> >>>>>> Thanks, >>>>> Hi Daniel, >>>>> >>>>> there is indeed a memory profiling patch under discussion: >>>>> http://gerrit.ovirt.org/#/c/32019/ >>>>> >>>>> but for your case we'll need a backport to 3.4.x and clearer i= nstall >>>>> instructions, >>>>> which I'll prepare as soon as possible. >>>> I updated the BZ (and are now blocking 54321/tcp on one of my h= osts). >>>> and verified it is not reachable. As general info: This system= I am >>>> using is my LAB / Test / eval setup for a final deployment for = ovirt >>>> (then 3.5) in production; so it will go away some time in the f= uture (a >>>> few weeks / months). If I am the only one experiencing this pro= blem >>>> then >>>> you might be better of allocating resources elsewhere ;) >>> Thanks for your understanding :) >>> >>> Unfortunately it is true that developer resources aren't so abun= dant, >>> but it is also true that memleaks should never be discarded easi= ly and >>> without >>> due investigation, considering the nature and the role of VDSM. >>> >>> So, I'm all in for further investigation regarding this issue. >>> >>>>> As for your question: if I understood correctly what you are a= sking >>>>> (still catching up the thread), if you are trying to rule out =
--=-XvEL1V2WgW8mfEoJpyaj Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Sandro, On Di, 2014-09-30 at 17:09 +0200, Sandro Bonazzola wrote: the
>>>>> stats >>>>> polling >>>>> made by Engine to this bad leak, one simple way to test is jus= t to >>>>> shutdown >>>>> Engine, >>>>> and let VDSMs run unguarded on hypervisors. You'll be able to = command >>>>> these >>>>> VDSMs using vdsClient or restarting Engine. >>>> As I said in my BZ comment this is not an option right now, but= if >>>> understand the matter correctly IPTABLES reject should ultimate= ly do >>>> the >>>> same? >>> Definitely yes! Just do whatever it is more convenient for you. >>> >> As you might have already seen in the BZ comment the leak stopped= after >> blocking the port. Though this is clearly no permanent option - p= lease >> let >> me know if I can be of any more assistance! > The immediate suspect in this situation is M2Crypto. Could you ver= ify > that by re-opening the firewall and setting ssl=3DFalse in vdsm.co= nf? > > You should disable ssl on Engine side and restart both Engine and = Vdsm > (too bad I do not recall how that's done on Engine: Piotr, can you= help?). > In vdc_options table there is option EncryptHostCommunication. Please confirm the following procedure is correct:
1. Change Postgres table value: # sudo -u postgres psql -U postgres engine -c "update vdc_options se= t option_value =3D 'false' where option_name =3D 'EncryptHostCommunica= tion';" engine=3D# SELECT * from vdc_options where option_name=3D'EncryptHostCommunication'; option_id | option_name | option_value | version -----------+--------------------------+--------------+--------- 335 | EncryptHostCommunication | false | general (1 row)
2. Restart engine 3. On the hosts; grep ssl /etc/vdsm/vdsm.conf #ssl =3D true ssl =3D false
4. restart VDSM
I assume I have to set 'ssl =3D false' this on on all hosts?
Please to set it to false and restart the engine.
I believe that you need to update a bit more on vdsm side. Please follow [1] section "Configure ovirt-engine and vdsm to work in= non-secure mode"
There is wrong name of the option and it should be EncryptHostCommuni= cation.
[1] http://www.ovirt.org/Developers_All_In_One I forgot; I suppose hosted-engine-ha is out of order because of disabl= ed ssl? =20 Indeed. And in hosted-engine, too, I need someone else's help (Sando?) to tell how to disable ssl. =20 in /etc/ovirt-hosted-engine: hosted-engine.conf just change: vdsm_use_ssl=3Dtrue to vdsm_use_ssl=3Dfalse
thanks. As turning off SSL fixes my immediate issue I will keep this off for now.
=20 =20 =20
=20
hosted-engine --connect-storage Connecting Storage Server Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2578, in <module> code, message =3D commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClient.py", line 712, in connectStorageServ= er res =3D self.s.connectStorageServer(serverType, spUUID, conList) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=3Dself.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request self.send_content(h, request_body) File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content connection.endheaders() File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders self._send_output() File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output self.send(msg) File "/usr/lib64/python2.6/httplib.py", line 739, in send self.connect() File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py"= , line 195, in connect cert_reqs=3Dself.cert_reqs) File "/usr/lib64/python2.6/ssl.py", line 342, in wrap_socket suppress_ragged_eofs=3Dsuppress_ragged_eofs) File "/usr/lib64/python2.6/ssl.py", line 120, in __init__ self.do_handshake() File "/usr/lib64/python2.6/ssl.py", line 279, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 8] _ssl.c:492: EOF occurred in violation of protocol =20 =20
--=20 Daniel Helgenberger=20 m box bewegtbild GmbH=20 P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19=20 D-10115 BERLIN=20 www.m-box.de www.monkeymen.tv=20 Gesch=C3=A4ftsf=C3=BChrer: Martin Retschitzegger / Michaela G=C3=B6llner Handeslregister: Amtsgericht Charlottenburg / HRB 112767=20 --=-XvEL1V2WgW8mfEoJpyaj Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIINtjCCBBYw ggL+oAMCAQICCwQAAAAAAS9O4S9SMA0GCSqGSIb3DQEBBQUAMFcxCzAJBgNVBAYTAkJFMRkwFwYD VQQKExBHbG9iYWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT aWduIFJvb3QgQ0EwHhcNMTEwNDEzMTAwMDAwWhcNMTkwNDEzMTAwMDAwWjBUMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQZXJzb25h bFNpZ24gMiBDQSAtIEcyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwWtB+TXs+BJ9 3SJRaV+3uRNGJ3cUO+MTgW8+5HQXfgy19CzkDI1T1NwwICi/bo4R/mYR5FEWx91//eE0ElC/89iY 7GkL0tDasmVx4TOXnrqrsziUcxEPPqHRE8x4NhtBK7+8o0nsMIJMA1gyZ2FA5To2Ew1BBuvovvDJ +Nua3qOCNBNu+8A+eNpJlVnlu/qB7+XWaPXtUMlsIikxD+gREFVUgYE4VzBuLa2kkg0VLd09XkE2 ceRDm6YgRATuDk6ogUyX4OLxCGIJF8yi6Z37M0wemDA6Uff0EuqdwDQd5HwG/rernUjt1grLdAxq 8BwywRRg0eFHmE+ShhpyO3Fi+wIDAQABo4HlMIHiMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8E CDAGAQH/AgEAMB0GA1UdDgQWBBQ/FdJtfC/nMZ5DCgaolGwsO8XuZTBHBgNVHSAEQDA+MDwGBFUd IAAwNDAyBggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8w MwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5nbG9iYWxzaWduLm5ldC9yb290LmNybDAfBgNV HSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0BAQUFAAOCAQEAQ3N5zKTMSTED HGFAgd/gu91Kb8AxPHgjq+7dhf7mkCinMqqrLai2XOrz8CP63BPaAx7oGOUBI0MyASBGk5zej9L3 oHtiF2BL01m1sBnT8rQxT2CJd/+jqjUl0p2ew8T3HSyatrsooGvDwf00yCB2JHTNvtQxNO8t6x/+ 048A1Q+0i7uf0nTnyrJLjD04zhL89ytetZspltOpJVYbmwiFjq6PxsdUNthUDme/9pOLmKDnQU0p W/JEwLs2TYCBNKwdgSGAk8/z+s2SCltKIG0Uh5U6t6j7JPuwNP/znImwMrlHDJ1YpW0rkF2PGraV CgDBf9dOB+IIpnwHfIi+LD+eITCCBMowggOyoAMCAQICEQCWaWbA3qWpL+Qmn6I16DynMA0GCSqG SIb3DQEBBQUAMFQxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYD VQQDEyFHbG9iYWxTaWduIFBlcnNvbmFsU2lnbiAyIENBIC0gRzIwHhcNMTMwODI3MTY1NzU4WhcN MTYwODI3MTY1NzU4WjBYMQswCQYDVQQGEwJERTEcMBoGA1UEAxMTRGFuaWVsIEhlbGdlbmJlcmdl cjErMCkGCSqGSIb3DQEJARYcZGFuaWVsLmhlbGdlbmJlcmdlckBtLWJveC5kZTCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBAM4BQ5vPknk1OGLd1qKSUIKmQLrjccjJcYj7qtAtA+fNYKF8 9p1VY4UwiFcF9jKlmA9Q8o8tYSx16LYYFoGWokNRAeKFXZiBZiHyI0ekpEfxo8N5cTMCcxKcSYWV 8sqzmBPCoMNpmiVoC8ec8Nv5SqXH34VVtDmNLfiVlsTyomBXAJkJ2/n5XqJzPLFGWWREtPLkVVS+ u426vt/hNsQi5akNoidYeXo98JcrmeApFJ3zB2KxvMziHx8LD4q1gAl9NumtX5YLbCpdWL9AkWdX Oaro3D9zj6Q6LyGwa/UQUrZdg3BXc07hjHZn6d9vet1SzpbyqQpTzM63yXiX1meEMlMCAwEAAaOC AZEwggGNMA4GA1UdDwEB/wQEAwIFoDBMBgNVHSAERTBDMEEGCSsGAQQBoDIBKDA0MDIGCCsGAQUF BwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAnBgNVHREEIDAegRxk YW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94LmRlMAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUH AwIGCCsGAQUFBwMEMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20v Z3MvZ3NwZXJzb25hbHNpZ24yZzIuY3JsMFUGCCsGAQUFBwEBBEkwRzBFBggrBgEFBQcwAoY5aHR0 cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NwZXJzb25hbHNpZ24yZzIuY3J0MB0G A1UdDgQWBBS8NFA/upd+Wipw2nj8RD/Ct+R2GTAfBgNVHSMEGDAWgBQ/FdJtfC/nMZ5DCgaolGws O8XuZTANBgkqhkiG9w0BAQUFAAOCAQEAXVTpu4fhOLETAW0zdbQiIwBIMZgeVNJnWV3GsMxByycU 63P+WBQTBl9qj47vHLmVdeF7MzH0QSXZSc9Tnfr6CYIImpyIZxRAGpAsWmtZf3JieRA0+j4GQJF2 zAea1NXYXoG9+ZSSZHBSxKUdrRdVdE320nuVGTT2HjEI2LEYbOvaXyi6HhpuHUiyu4LD0+RIT3fi T8jUiKKLTsApTD+Ak8SLF0IESOSA6htirv69mDDC7Klg9dT7QBPO7dpoKIUOldV3VhahndVfsDff KD7pkUUvG5XftYEQOxlWDJzuTBeqf/4hxXMtzFU9OaI6oKJjLfr6B+XBc6xwOtc/NMWmejCCBMow ggOyoAMCAQICEQCWaWbA3qWpL+Qmn6I16DynMA0GCSqGSIb3DQEBBQUAMFQxCzAJBgNVBAYTAkJF MRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYDVQQDEyFHbG9iYWxTaWduIFBlcnNvbmFs U2lnbiAyIENBIC0gRzIwHhcNMTMwODI3MTY1NzU4WhcNMTYwODI3MTY1NzU4WjBYMQswCQYDVQQG EwJERTEcMBoGA1UEAxMTRGFuaWVsIEhlbGdlbmJlcmdlcjErMCkGCSqGSIb3DQEJARYcZGFuaWVs LmhlbGdlbmJlcmdlckBtLWJveC5kZTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM4B Q5vPknk1OGLd1qKSUIKmQLrjccjJcYj7qtAtA+fNYKF89p1VY4UwiFcF9jKlmA9Q8o8tYSx16LYY FoGWokNRAeKFXZiBZiHyI0ekpEfxo8N5cTMCcxKcSYWV8sqzmBPCoMNpmiVoC8ec8Nv5SqXH34VV tDmNLfiVlsTyomBXAJkJ2/n5XqJzPLFGWWREtPLkVVS+u426vt/hNsQi5akNoidYeXo98JcrmeAp FJ3zB2KxvMziHx8LD4q1gAl9NumtX5YLbCpdWL9AkWdXOaro3D9zj6Q6LyGwa/UQUrZdg3BXc07h jHZn6d9vet1SzpbyqQpTzM63yXiX1meEMlMCAwEAAaOCAZEwggGNMA4GA1UdDwEB/wQEAwIFoDBM BgNVHSAERTBDMEEGCSsGAQQBoDIBKDA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxz aWduLmNvbS9yZXBvc2l0b3J5LzAnBgNVHREEIDAegRxkYW5pZWwuaGVsZ2VuYmVyZ2VyQG0tYm94 LmRlMAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMEMGA1UdHwQ8MDow OKA2oDSGMmh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vZ3MvZ3NwZXJzb25hbHNpZ24yZzIuY3Js MFUGCCsGAQUFBwEBBEkwRzBFBggrBgEFBQcwAoY5aHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNv bS9jYWNlcnQvZ3NwZXJzb25hbHNpZ24yZzIuY3J0MB0GA1UdDgQWBBS8NFA/upd+Wipw2nj8RD/C t+R2GTAfBgNVHSMEGDAWgBQ/FdJtfC/nMZ5DCgaolGwsO8XuZTANBgkqhkiG9w0BAQUFAAOCAQEA XVTpu4fhOLETAW0zdbQiIwBIMZgeVNJnWV3GsMxByycU63P+WBQTBl9qj47vHLmVdeF7MzH0QSXZ Sc9Tnfr6CYIImpyIZxRAGpAsWmtZf3JieRA0+j4GQJF2zAea1NXYXoG9+ZSSZHBSxKUdrRdVdE32 0nuVGTT2HjEI2LEYbOvaXyi6HhpuHUiyu4LD0+RIT3fiT8jUiKKLTsApTD+Ak8SLF0IESOSA6hti rv69mDDC7Klg9dT7QBPO7dpoKIUOldV3VhahndVfsDffKD7pkUUvG5XftYEQOxlWDJzuTBeqf/4h xXMtzFU9OaI6oKJjLfr6B+XBc6xwOtc/NMWmejGCAucwggLjAgEBMGkwVDELMAkGA1UEBhMCQkUx GTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24gUGVyc29uYWxT aWduIDIgQ0EgLSBHMgIRAJZpZsDepakv5CafojXoPKcwCQYFKw4DAhoFAKCCAVMwGAYJKoZIhvcN AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTQwOTMwMTU1NDIyWjAjBgkqhkiG9w0B CQQxFgQUdJMJUVgVrY4rfXySAfifOW8vuOYweAYJKwYBBAGCNxAEMWswaTBUMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQZXJzb25h bFNpZ24gMiBDQSAtIEcyAhEAlmlmwN6lqS/kJp+iNeg8pzB6BgsqhkiG9w0BCRACCzFroGkwVDEL MAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNp Z24gUGVyc29uYWxTaWduIDIgQ0EgLSBHMgIRAJZpZsDepakv5CafojXoPKcwDQYJKoZIhvcNAQEB BQAEggEAkDQGFZBygXyDAFtgEx1MiPOLvh+Uxwh1EpRqi5d2bmHidjIsDdK33ZYkGi8OL0zIHcKr a+eaSyDeWGS517yoSpOmRP7Ojy+gihZuzDYHp0qbx8IDDMyy7cW9wgzkgWC1Uuoh7M2wVbmeLAe2 r4N2cHZQxFWQNZcPxwRQXEAoGRDQDoUkCNKZsz4Rtz87Mr/efkZ31PvKQneNHpBSZYl6C0uEWAaJ AKk6rDxFStNTdSRYkVx/9Ru5SLUUeChsomkSlprabew6wp3tX4bc4xRd+NBogESY3yul/a2M45zW bccTvurhPp9v3rkCXbSm1GuyDkohN6x1Iy5jrzrhYOizngAAAAAAAA== --=-XvEL1V2WgW8mfEoJpyaj--