Re: [ovirt-users] SSLHandshakeException: Received fatal alert: certificate_expired

Thanks for the guidance everyone. I've upgraded my engine now to ovirt-engine-3.4.4-1 but I've still got the same error unfortunately. Below is the output of the upgrade. Should this have fixed the issue or do I need to upgrade to 3.5 etc?

[ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup- 20170922125526-vw5khx.log Version: otopi-1.2.3 (otopi-1.2.3-1.el6) [ INFO ] Stage: Environment packages setup [ INFO ] Yum Downloading: repomdPLa0LXtmp.xml (0%) [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- --== PACKAGES ==-- [ INFO ] Checking for product updates... Setup has found updates for some packages, do you wish to update them now? (Yes, No) [Yes]: [ INFO ] Checking for an update for Setup... --== NETWORK CONFIGURATION ==-- [WARNING] Failed to resolve engine01.mydomain.za using DNS, it can be resolved only locally Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. Do you want Setup to configure the firewall? (Yes, No) [Yes]: no --== DATABASE CONFIGURATION ==-- --== OVIRT ENGINE CONFIGURATION ==-- Skipping storing options as database already prepared --== PKI CONFIGURATION ==-- PKI is already configured --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [WARNING] Less than 16384MB of memory is available [ INFO ] Cleaning stale zombie tasks --== CONFIGURATION PREVIEW ==-- Engine database name : engine Engine database secured connection : False Engine database host : localhost Engine database user name : engine Engine database host name validation : False Engine database port : 5432 Datacenter storage type : False Update Firewall : False Configure WebSocket Proxy : True Host FQDN : engine01.mydomain.za Upgrade packages : True Please confirm installation settings (OK, Cancel) [OK]: [ INFO ] Cleaning async tasks and compensations [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Yum Status: Downloading Packages [ INFO ] Yum Download/Verify: ovirt-engine-3.4.4-1.el6.noarch [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 2.0 M(19%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 4.3 M(41%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 6.3 M(60%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 8.9 M(85%) [ INFO ] Yum Download/Verify: ovirt-engine-backend-3.4.4-1.el6.noarch [ INFO ] Yum Download/Verify: ovirt-engine-dbscripts-3.4.4-1.el6.noarch (I've taken out all the downloading progress) [ INFO ] Yum Verify: 26/26: ovirt-engine-backend.noarch 0:3.4.0-1.el6 - ud [ INFO ] Stage: Misc configuration [ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/ backups/engine-20170922143709.m_8fr_.dump'. [ INFO ] Updating Engine database schema [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up --== SUMMARY ==-- [WARNING] Less than 16384MB of memory is available SSH fingerprint: 86:C7:AA:35:45:E9:83:3E:16:C9:2A:F5:68:52:68:84 Internal CA EE:91:B3:E7:40:D7:DD:A7:DD:77: 9C:3B:D5:A1:E7:BE:E2:C9:8B:AA Web access is enabled at: http://engine01.mydomain.za:80/ovirt-engine https://engine01.mydomain.za:443/ovirt-engine In order to configure firewalld, copy the files from /etc/ovirt-engine/firewalld to /etc/firewalld/services and execute the following commands: firewall-cmd -service ovirt-postgres firewall-cmd -service ovirt-https firewall-cmd -service ovirt-websocket-proxy firewall-cmd -service ovirt-http The following network ports should be opened: tcp:443 tcp:5432 tcp:6100 tcp:80 An example of the required configuration for iptables can be found at: /etc/ovirt-engine/iptables.example --== END OF SUMMARY ==-- [ INFO ] Starting engine service [ INFO ] Restarting httpd [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ ovirt-engine-setup-20170922125526-vw5khx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/ answers/20170922143806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully I'm still seeing the following below, in my engine.log and when I log in, all my VM's show as unknown. 2017-09-22 15:06:06,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za,d2debdfe-76e7-40cf-a7fd-78a0f50f14d4]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired Any ideas? Thanks! On Fri, Sep 22, 2017 at 11:10 AM, Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123(a)gmail.com&gt; wrote: > >> Thanks Martin and Piotr, >> >> Correct, this was a very old installation from the old drey repo that >> was upgraded gradually over the years. >> >> I have tried engine-setup yesterday, prior to this looking under >> /var/log/ovirt-engine/setup it looks like 2014 >> >> I've attached a log of the output of running it now, looks like a repo >> issue with trying to upgrade to the latest 3.4.x release, but not sure what >> else to look for? >> > > ​Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not > working anymore. You can either: > > 1. Execute engine-setup --offline to skip updates check or > 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to > main site resources.ovirt.org > > >> Thanks for the assistance. >> >> Regards. >> >> Neil Wilson >> >> >> On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < >> piotr.kliczewski(a)gmail.com&gt; wrote: >> >>> On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina(a)redhat.com&gt; >>> wrote: >>> > >>> > >>> > On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123(a)gmail.com&gt; wrote: >>> >> >>> >> Hi Piotr, >>> >> >>> >> Thank you for the information. >>> >> >>> >> It looks like something has expired looking in the server.log now >>> that >>> >> debug is enabled. >>> >> >>> >> 2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) >>> Version: >>> >> V3 >>> >> 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) >>> Subject: >>> >> CN=engine01.mydomain.za, O=mydomain, C=US >>> >> 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) >>> >> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 >>> >> 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) >>> >> 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) >>> Key: >>> >> Sun RSA public key, 1024 bits >>> >> 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) >>> modulus: >>> >> 966706131850237857720016566132274169225143716493132034132811 >>> 213711757321195965137528821713060454503460188878350322233731 >>> 259812207539722762942035931744044702655933680916835641105243 >>> 164032601213316092139626126181817086803318505413903188689260 >>> 54438078223371655800890725486783860059873397983318033852172060923531 >>> >> 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) >>> public >>> >> exponent: 65537 >>> >> 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) >>> >> Validity: [From: Sun Oct 14 22:26:46 SAST 2012, >>> >> 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) >>> >> To: Tue Sep 19 18:26:49 SAST 2017] >>> >> 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) >>> Issuer: >>> >> CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US >>> >> >>> >> Any idea how I can generate a new one and what cert it is that's >>> expired? >>> > >>> > >>> > It seems that your engine certificate has expired, but AFAIK this >>> > certificate should be automatically renewed during engine-setup. So >>> when did >>> > you execute engine-setup for last time? Any info/warning about this >>> shown >>> > during invocation? >>> >>> Correct, Martin was a bit faster then me :) >>> >>> > >>> > Also looking at server.log I found JBoss 7.1.1, so you are using >>> really >>> > ancient oVirt, version, right? >>> > >>> >> >>> >> Please see the attached log for more info. >>> >> >>> >> Thank you so much for your assistance. >>> >> >>> >> Regards. >>> >> >>> >> Neil Wilson. >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski >>> >> <piotr.kliczewski(a)gmail.com&gt; wrote: >>> >>> >>> >>> Neil, >>> >>> >>> >>> It seems that your engine certificate(s) is/are not ok. I would >>> >>> suggest to enable ssl debug in the engine by: >>> >>> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. >>> >>> - restart your engine >>> >>> - check your server.log and check what is the issue. >>> >>> >>> >>> Hopefully we will be able to understand what happened in your setup. >>> >>> >>> >>> Thanks, >>> >>> Piotr >>> >>> >>> >>> [1] >>> >>> https://github.com/oVirt/ovirt-engine/blob/master/packaging/ >>> services/ovirt-engine/ovirt-engine.py#L341 >>> >>> >>> >>> On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123(a)gmail.com&gt; wrote: >>> >>> > Further to the logs sent, on the nodes I'm also seeing the >>> following >>> >>> > error >>> >>> > under /var/log/messages... >>> >>> > >>> >>> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate >>> with >>> >>> > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C >>> >>> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler >>> >>> > exception#012Traceback >>> >>> > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC >>> .py", >>> >>> > line >>> >>> > 80, in threaded_start#012 self.server.handle_request()#012 >>> File >>> >>> > "/usr/lib64/python2.6/SocketServer.py", line 278, in >>> handle_request#012 >>> >>> > self._handle_request_noblock()#012 File >>> >>> > "/usr/lib64/python2.6/SocketServer.py", line 288, in >>> >>> > _handle_request_noblock#012 request, client_address = >>> >>> > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe >>> rver.py", >>> >>> > line >>> >>> > 456, in get_request#012 return self.socket.accept()#012 File >>> >>> > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", >>> line >>> >>> > 136, >>> >>> > in accept#012 raise SSL.SSLError("%s, client %s" % (e, >>> >>> > address[0]))#012SSLError: no certificate returned, client >>> 10.251.193.5 >>> >>> > >>> >>> > Not sure if this is any further help in diagnosing the issue? >>> >>> > >>> >>> > Thanks, any assistance is appreciated. >>> >>> > >>> >>> > Regards. >>> >>> > >>> >>> > Neil Wilson. >>> >>> > >>> >>> > >>> >>> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123(a)gmail.com&gt; >>> wrote: >>> >>> >> >>> >>> >> Hi Piotr, >>> >>> >> >>> >>> >> Thank you for the reply. After sending the email I did go and >>> check >>> >>> >> the >>> >>> >> engine one too.... >>> >>> >> >>> >>> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem >>> >>> >> -enddate >>> >>> >> -noout >>> >>> >> notAfter=Oct 13 16:26:46 2022 GMT >>> >>> >> >>> >>> >> I'm not sure if this one below is meant to verify or if this >>> output is >>> >>> >> expected? >>> >>> >> >>> >>> >> [root@engine01 /]# openssl x509 -in >>> >>> >> /etc/pki/ovirt-engine/private/ca.pem >>> >>> >> -enddate -noout >>> >>> >> unable to load certificate >>> >>> >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no >>> start >>> >>> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE >>> >>> >> >>> >>> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017 >>> >>> >> >>> >>> >> Any ideas? >>> >>> >> >>> >>> >> Googling surprisingly doesn't come up with much. >>> >>> >> >>> >>> >> Thank you. >>> >>> >> >>> >>> >> Regards. >>> >>> >> >>> >>> >> Neil Wilson. >>> >>> >> >>> >>> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski >>> >>> >> <piotr.kliczewski(a)gmail.com&gt; wrote: >>> >>> >>> >>> >>> >>> Neil, >>> >>> >>> >>> >>> >>> You checked both nodes what about the engine? Can you check >>> engine >>> >>> >>> certs? >>> >>> >>> You can find more info where they are located here [1]. >>> >>> >>> >>> >>> >>> Thanks, >>> >>> >>> Piotr >>> >>> >>> >>> >>> >>> [1] >>> >>> >>> >>> >>> >>> https://www.ovirt.org/develop/release-management/features/in >>> fra/pki/#ovirt-engine >>> >>> >>> >>> >>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123(a)gmail.com&gt; >>> wrote: >>> >>> >>> > Hi guys, >>> >>> >>> > >>> >>> >>> > Please could someone assist, my cluster is down and I can't >>> access >>> >>> >>> > my >>> >>> >>> > vm's >>> >>> >>> > to switch some of them back on. >>> >>> >>> > >>> >>> >>> > I'm seeing the following error in the engine.log however I've >>> >>> >>> > checked >>> >>> >>> > my >>> >>> >>> > certs on my hosts (as some of the goolge results said to >>> check), >>> >>> >>> > but >>> >>> >>> > the >>> >>> >>> > certs haven't expired... >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > 2017-09-21 15:09:45,077 ERROR >>> >>> >>> > >>> >>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >>> SCommand] >>> >>> >>> > (DefaultQuartzScheduler_Worker-4) Command >>> >>> >>> > GetCapabilitiesVDSCommand(HostName >>> >>> >>> > = node02.mydomain.za, HostId = >>> >>> >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, >>> >>> >>> > vds=Host[node02.mydomain.za]) execution failed. Exception: >>> >>> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: >>> Received >>> >>> >>> > fatal >>> >>> >>> > alert: certificate_expired >>> >>> >>> > 2017-09-21 15:09:45,086 ERROR >>> >>> >>> > >>> >>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD >>> SCommand] >>> >>> >>> > (DefaultQuartzScheduler_Worker-10) Command >>> >>> >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, >>> HostId = >>> >>> >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ >>> node01.mydomain.za]) >>> >>> >>> > execution failed. Exception: VDSNetworkException: >>> >>> >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert: >>> >>> >>> > certificate_expired >>> >>> >>> > 2017-09-21 15:09:48,173 ERROR >>> >>> >>> > >>> >>> >>> > My engine and host info is below... >>> >>> >>> > >>> >>> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt >>> >>> >>> > ovirt-engine-lib-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch >>> >>> >>> > ovirt-engine-setup-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-host-deploy-1.2.0-1.el6.noarch >>> >>> >>> > ovirt-engine-backend-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-image-uploader-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-tools-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch >>> >>> >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch >>> >>> >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-log-collector-3.4.1-1.el6.noarch >>> >>> >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch >>> >>> >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.no >>> arch >>> >>> >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch >>> >>> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release >>> >>> >>> > CentOS release 6.5 (Final) >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > [root@node02 ~]# openssl x509 -in >>> /etc/pki/vdsm/certs/vdsmcert.pem >>> >>> >>> > -enddate >>> >>> >>> > -noout ; date >>> >>> >>> > notAfter=May 27 08:36:17 2019 GMT >>> >>> >>> > Thu Sep 21 15:18:22 SAST 2017 >>> >>> >>> > CentOS release 6.5 (Final) >>> >>> >>> > [root@node02 ~]# rpm -qa | grep vdsm >>> >>> >>> > vdsm-4.14.6-0.el6.x86_64 >>> >>> >>> > vdsm-python-4.14.6-0.el6.x86_64 >>> >>> >>> > vdsm-cli-4.14.6-0.el6.noarch >>> >>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >>> >>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > [root@node01 ~]# openssl x509 -in >>> /etc/pki/vdsm/certs/vdsmcert.pem >>> >>> >>> > -enddate >>> >>> >>> > -noout ; date >>> >>> >>> > notAfter=Jun 13 16:09:41 2018 GMT >>> >>> >>> > Thu Sep 21 15:18:52 SAST 2017 >>> >>> >>> > CentOS release 6.5 (Final) >>> >>> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm >>> >>> >>> > vdsm-4.14.6-0.el6.x86_64 >>> >>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >>> >>> >>> > vdsm-cli-4.14.6-0.el6.noarch >>> >>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >>> >>> >>> > vdsm-python-4.14.6-0.el6.x86_64 >>> >>> >>> > >>> >>> >>> > Please could I have some assistance, I'm rater desperate. >>> >>> >>> > >>> >>> >>> > Thank you. >>> >>> >>> > >>> >>> >>> > Regards. >>> >>> >>> > >>> >>> >>> > Neil Wilson >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > >>> >>> >>> > _______________________________________________ >>> >>> >>> > Users mailing list >>> >>> >>> > Users(a)ovirt.org >>> >>> >>> > http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> > >>> >>> >> >>> >>> >> >>> >>> > >>> >> >>> >> >>> >> >>> >> _______________________________________________ >>> >> Users mailing list >>> >> Users(a)ovirt.org >>> >> http://lists.ovirt.org/mailman/listinfo/users >>> >> >>> > >>> >> >> >