Re: [ovirt-users] SSLHandshakeException: Received fatal alert: certificate_expired

On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123(a)gmail.com&gt; wrote: > Thanks Martin and Piotr, > > Correct, this was a very old installation from the old drey repo that was > upgraded gradually over the years. > > I have tried engine-setup yesterday, prior to this looking under > /var/log/ovirt-engine/setup it looks like 2014 > > I've attached a log of the output of running it now, looks like a repo > issue with trying to upgrade to the latest 3.4.x release, but not sure what > else to look for? > ​Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either: 1. Execute engine-setup --offline to skip updates check or 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org > Thanks for the assistance. > > Regards. > > Neil Wilson > > > On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < > piotr.kliczewski(a)gmail.com&gt; wrote: > >> On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> > >> > >> > On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123(a)gmail.com&gt; wrote: >> >> >> >> Hi Piotr, >> >> >> >> Thank you for the information. >> >> >> >> It looks like something has expired looking in the server.log now that >> >> debug is enabled. >> >> >> >> 2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) >> Version: >> >> V3 >> >> 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) >> Subject: >> >> CN=engine01.mydomain.za, O=mydomain, C=US >> >> 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) >> >> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 >> >> 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) >> >> 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: >> >> Sun RSA public key, 1024 bits >> >> 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) >> modulus: >> >> 966706131850237857720016566132274169225143716493132034132811 >> 213711757321195965137528821713060454503460188878350322233731 >> 259812207539722762942035931744044702655933680916835641105243 >> 164032601213316092139626126181817086803318505413903188689260 >> 54438078223371655800890725486783860059873397983318033852172060923531 >> >> 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) >> public >> >> exponent: 65537 >> >> 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) >> >> Validity: [From: Sun Oct 14 22:26:46 SAST 2012, >> >> 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) >> >> To: Tue Sep 19 18:26:49 SAST 2017] >> >> 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) >> Issuer: >> >> CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US >> >> >> >> Any idea how I can generate a new one and what cert it is that's >> expired? >> > >> > >> > It seems that your engine certificate has expired, but AFAIK this >> > certificate should be automatically renewed during engine-setup. So >> when did >> > you execute engine-setup for last time? Any info/warning about this >> shown >> > during invocation? >> >> Correct, Martin was a bit faster then me :) >> >> > >> > Also looking at server.log I found JBoss 7.1.1, so you are using really >> > ancient oVirt, version, right? >> > >> >> >> >> Please see the attached log for more info. >> >> >> >> Thank you so much for your assistance. >> >> >> >> Regards. >> >> >> >> Neil Wilson. >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski >> >> <piotr.kliczewski(a)gmail.com&gt; wrote: >> >>> >> >>> Neil, >> >>> >> >>> It seems that your engine certificate(s) is/are not ok. I would >> >>> suggest to enable ssl debug in the engine by: >> >>> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. >> >>> - restart your engine >> >>> - check your server.log and check what is the issue. >> >>> >> >>> Hopefully we will be able to understand what happened in your setup. >> >>> >> >>> Thanks, >> >>> Piotr >> >>> >> >>> [1] >> >>> https://github.com/oVirt/ovirt-engine/blob/master/packaging/ >> services/ovirt-engine/ovirt-engine.py#L341 >> >>> >> >>> On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123(a)gmail.com&gt; wrote: >> >>> > Further to the logs sent, on the nodes I'm also seeing the >> following >> >>> > error >> >>> > under /var/log/messages... >> >>> > >> >>> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate >> with >> >>> > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C >> >>> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler >> >>> > exception#012Traceback >> >>> > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC >> .py", >> >>> > line >> >>> > 80, in threaded_start#012 self.server.handle_request()#012 >> File >> >>> > "/usr/lib64/python2.6/SocketServer.py", line 278, in >> handle_request#012 >> >>> > self._handle_request_noblock()#012 File >> >>> > "/usr/lib64/python2.6/SocketServer.py", line 288, in >> >>> > _handle_request_noblock#012 request, client_address = >> >>> > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe >> rver.py", >> >>> > line >> >>> > 456, in get_request#012 return self.socket.accept()#012 File >> >>> > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", >> line >> >>> > 136, >> >>> > in accept#012 raise SSL.SSLError("%s, client %s" % (e, >> >>> > address[0]))#012SSLError: no certificate returned, client >> 10.251.193.5 >> >>> > >> >>> > Not sure if this is any further help in diagnosing the issue? >> >>> > >> >>> > Thanks, any assistance is appreciated. >> >>> > >> >>> > Regards. >> >>> > >> >>> > Neil Wilson. >> >>> > >> >>> > >> >>> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123(a)gmail.com&gt; >> wrote: >> >>> >> >> >>> >> Hi Piotr, >> >>> >> >> >>> >> Thank you for the reply. After sending the email I did go and >> check >> >>> >> the >> >>> >> engine one too.... >> >>> >> >> >>> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem >> >>> >> -enddate >> >>> >> -noout >> >>> >> notAfter=Oct 13 16:26:46 2022 GMT >> >>> >> >> >>> >> I'm not sure if this one below is meant to verify or if this >> output is >> >>> >> expected? >> >>> >> >> >>> >> [root@engine01 /]# openssl x509 -in >> >>> >> /etc/pki/ovirt-engine/private/ca.pem >> >>> >> -enddate -noout >> >>> >> unable to load certificate >> >>> >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start >> >>> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE >> >>> >> >> >>> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017 >> >>> >> >> >>> >> Any ideas? >> >>> >> >> >>> >> Googling surprisingly doesn't come up with much. >> >>> >> >> >>> >> Thank you. >> >>> >> >> >>> >> Regards. >> >>> >> >> >>> >> Neil Wilson. >> >>> >> >> >>> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski >> >>> >> <piotr.kliczewski(a)gmail.com&gt; wrote: >> >>> >>> >> >>> >>> Neil, >> >>> >>> >> >>> >>> You checked both nodes what about the engine? Can you check >> engine >> >>> >>> certs? >> >>> >>> You can find more info where they are located here [1]. >> >>> >>> >> >>> >>> Thanks, >> >>> >>> Piotr >> >>> >>> >> >>> >>> [1] >> >>> >>> >> >>> >>> https://www.ovirt.org/develop/release-management/features/in >> fra/pki/#ovirt-engine >> >>> >>> >> >>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123(a)gmail.com&gt; >> wrote: >> >>> >>> > Hi guys, >> >>> >>> > >> >>> >>> > Please could someone assist, my cluster is down and I can't >> access >> >>> >>> > my >> >>> >>> > vm's >> >>> >>> > to switch some of them back on. >> >>> >>> > >> >>> >>> > I'm seeing the following error in the engine.log however I've >> >>> >>> > checked >> >>> >>> > my >> >>> >>> > certs on my hosts (as some of the goolge results said to >> check), >> >>> >>> > but >> >>> >>> > the >> >>> >>> > certs haven't expired... >> >>> >>> > >> >>> >>> > >> >>> >>> > 2017-09-21 15:09:45,077 ERROR >> >>> >>> > >> >>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >> SCommand] >> >>> >>> > (DefaultQuartzScheduler_Worker-4) Command >> >>> >>> > GetCapabilitiesVDSCommand(HostName >> >>> >>> > = node02.mydomain.za, HostId = >> >>> >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, >> >>> >>> > vds=Host[node02.mydomain.za]) execution failed. Exception: >> >>> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: >> Received >> >>> >>> > fatal >> >>> >>> > alert: certificate_expired >> >>> >>> > 2017-09-21 15:09:45,086 ERROR >> >>> >>> > >> >>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >> SCommand] >> >>> >>> > (DefaultQuartzScheduler_Worker-10) Command >> >>> >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, >> HostId = >> >>> >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ >> node01.mydomain.za]) >> >>> >>> > execution failed. Exception: VDSNetworkException: >> >>> >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert: >> >>> >>> > certificate_expired >> >>> >>> > 2017-09-21 15:09:48,173 ERROR >> >>> >>> > >> >>> >>> > My engine and host info is below... >> >>> >>> > >> >>> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt >> >>> >>> > ovirt-engine-lib-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch >> >>> >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch >> >>> >>> > ovirt-engine-setup-3.4.0-1.el6.noarch >> >>> >>> > ovirt-host-deploy-1.2.0-1.el6.noarch >> >>> >>> > ovirt-engine-backend-3.4.0-1.el6.noarch >> >>> >>> > ovirt-image-uploader-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-tools-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch >> >>> >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch >> >>> >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch >> >>> >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch >> >>> >>> > ovirt-log-collector-3.4.1-1.el6.noarch >> >>> >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch >> >>> >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.no >> arch >> >>> >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch >> >>> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release >> >>> >>> > CentOS release 6.5 (Final) >> >>> >>> > >> >>> >>> > >> >>> >>> > [root@node02 ~]# openssl x509 -in >> /etc/pki/vdsm/certs/vdsmcert.pem >> >>> >>> > -enddate >> >>> >>> > -noout ; date >> >>> >>> > notAfter=May 27 08:36:17 2019 GMT >> >>> >>> > Thu Sep 21 15:18:22 SAST 2017 >> >>> >>> > CentOS release 6.5 (Final) >> >>> >>> > [root@node02 ~]# rpm -qa | grep vdsm >> >>> >>> > vdsm-4.14.6-0.el6.x86_64 >> >>> >>> > vdsm-python-4.14.6-0.el6.x86_64 >> >>> >>> > vdsm-cli-4.14.6-0.el6.noarch >> >>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >> >>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >> >>> >>> > >> >>> >>> > >> >>> >>> > [root@node01 ~]# openssl x509 -in >> /etc/pki/vdsm/certs/vdsmcert.pem >> >>> >>> > -enddate >> >>> >>> > -noout ; date >> >>> >>> > notAfter=Jun 13 16:09:41 2018 GMT >> >>> >>> > Thu Sep 21 15:18:52 SAST 2017 >> >>> >>> > CentOS release 6.5 (Final) >> >>> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm >> >>> >>> > vdsm-4.14.6-0.el6.x86_64 >> >>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >> >>> >>> > vdsm-cli-4.14.6-0.el6.noarch >> >>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >> >>> >>> > vdsm-python-4.14.6-0.el6.x86_64 >> >>> >>> > >> >>> >>> > Please could I have some assistance, I'm rater desperate. >> >>> >>> > >> >>> >>> > Thank you. >> >>> >>> > >> >>> >>> > Regards. >> >>> >>> > >> >>> >>> > Neil Wilson >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > _______________________________________________ >> >>> >>> > Users mailing list >> >>> >>> > Users(a)ovirt.org >> >>> >>> > http://lists.ovirt.org/mailman/listinfo/users >> >>> >>> > >> >>> >> >> >>> >> >> >>> > >> >> >> >> >> >> >> >> _______________________________________________ >> >> Users mailing list >> >> Users(a)ovirt.org >> >> http://lists.ovirt.org/mailman/listinfo/users >> >> >> > >> > >