Thanks for the guidance everyone.

I've upgraded my engine now to ovirt-engine-3.4.4-1 but I've still got the same error unfortunately. Below is the output of the upgrade. Should this have fixed the issue or do I need to upgrade to 3.5 etc?

[ INFO  ] Stage: Initializing

[ INFO  ] Stage: Environment setup

          Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf']

          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170922125526-vw5khx.log

          Version: otopi-1.2.3 (otopi-1.2.3-1.el6)

[ INFO  ] Stage: Environment packages setup

[ INFO  ] Yum Downloading: repomdPLa0LXtmp.xml (0%)

[ INFO  ] Stage: Programs detection

[ INFO  ] Stage: Environment setup

[ INFO  ] Stage: Environment customization

         

          --== PRODUCT OPTIONS ==--

         

         

          --== PACKAGES ==--

         

[ INFO  ] Checking for product updates...

          Setup has found updates for some packages, do you wish to update them now? (Yes, No) [Yes]: 

[ INFO  ] Checking for an update for Setup...

         

          --== NETWORK CONFIGURATION ==--

         

[WARNING] Failed to resolve engine01.mydomain.za using DNS, it can be resolved only locally

          Setup can automatically configure the firewall on this system.

          Note: automatic configuration of the firewall may overwrite current settings.

          Do you want Setup to configure the firewall? (Yes, No) [Yes]: no

         

          --== DATABASE CONFIGURATION ==--

         

         

          --== OVIRT ENGINE CONFIGURATION ==--

         

          Skipping storing options as database already prepared

         

          --== PKI CONFIGURATION ==--

         

          PKI is already configured

         

          --== APACHE CONFIGURATION ==--

         

         

          --== SYSTEM CONFIGURATION ==--

         

         

          --== MISC CONFIGURATION ==--

         

         

          --== END OF CONFIGURATION ==--

         

[ INFO  ] Stage: Setup validation

          During execution engine service will be stopped (OK, Cancel) [OK]: 

[WARNING] Less than 16384MB of memory is available

[ INFO  ] Cleaning stale zombie tasks

         

          --== CONFIGURATION PREVIEW ==--

         

          Engine database name                    : engine

          Engine database secured connection      : False

          Engine database host                    : localhost

          Engine database user name               : engine

          Engine database host name validation    : False

          Engine database port                    : 5432

          Datacenter storage type                 : False

          Update Firewall                         : False

          Configure WebSocket Proxy               : True

          Host FQDN                               : engine01.mydomain.za

          Upgrade packages                        : True

         

          Please confirm installation settings (OK, Cancel) [OK]: 

[ INFO  ] Cleaning async tasks and compensations

[ INFO  ] Checking the Engine database consistency

[ INFO  ] Stage: Transaction setup

[ INFO  ] Stopping engine service

[ INFO  ] Stopping websocket-proxy service

[ INFO  ] Stage: Misc configuration

[ INFO  ] Stage: Package installation

[ INFO  ] Yum Status: Downloading Packages

[ INFO  ] Yum Download/Verify: ovirt-engine-3.4.4-1.el6.noarch

[ INFO  ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 2.0 M(19%)

[ INFO  ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 4.3 M(41%)

[ INFO  ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 6.3 M(60%)

[ INFO  ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 8.9 M(85%)

[ INFO  ] Yum Download/Verify: ovirt-engine-backend-3.4.4-1.el6.noarch

[ INFO  ] Yum Download/Verify: ovirt-engine-dbscripts-3.4.4-1.el6.noarch

(I've taken out all the downloading progress)

[ INFO  ] Yum Verify: 26/26: ovirt-engine-backend.noarch 0:3.4.0-1.el6 - ud

[ INFO  ] Stage: Misc configuration

[ INFO  ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20170922143709.m_8fr_.dump'.

[ INFO  ] Updating Engine database schema

[ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'

[ INFO  ] Stage: Transaction commit

[ INFO  ] Stage: Closing up

         

          --== SUMMARY ==--

         

[WARNING] Less than 16384MB of memory is available

          SSH fingerprint: 86:C7:AA:35:45:E9:83:3E:16:C9:2A:F5:68:52:68:84

          Internal CA EE:91:B3:E7:40:D7:DD:A7:DD:77:9C:3B:D5:A1:E7:BE:E2:C9:8B:AA

          Web access is enabled at:

              http://engine01.mydomain.za:80/ovirt-engine

              https://engine01.mydomain.za:443/ovirt-engine

          In order to configure firewalld, copy the files from

              /etc/ovirt-engine/firewalld to /etc/firewalld/services

              and execute the following commands:

              firewall-cmd -service ovirt-postgres

              firewall-cmd -service ovirt-https

              firewall-cmd -service ovirt-websocket-proxy

              firewall-cmd -service ovirt-http

          The following network ports should be opened:

              tcp:443

              tcp:5432

              tcp:6100

              tcp:80

          An example of the required configuration for iptables can be found at:

              /etc/ovirt-engine/iptables.example

         

          --== END OF SUMMARY ==--

         

[ INFO  ] Starting engine service

[ INFO  ] Restarting httpd

[ INFO  ] Stage: Clean up

          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170922125526-vw5khx.log

[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170922143806-setup.conf'

[ INFO  ] Stage: Pre-termination

[ INFO  ] Stage: Termination

[ INFO  ] Execution of setup completed successfully 

I'm still seeing the following below, in my engine.log and when I log in, all my VM's show as unknown.

2017-09-22 15:06:06,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za,d2debdfe-76e7-40cf-a7fd-78a0f50f14d4]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired

Any ideas?

Thanks!

On Fri, Sep 22, 2017 at 11:10 AM, Martin Perina <mperina@redhat.com> wrote:

On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123@gmail.com> wrote:

Thanks Martin and Piotr,

Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.

I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014

I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?

​Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either:

1. Execute engine-setup --offline to skip updates check or

2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org

Thanks for the assistance.

Regards.

Neil Wilson

On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:

On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
>
>
> On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
>>
>> Hi Piotr,
>>
>> Thank you for the information.
>>
>> It looks like something has expired looking in the server.log now that
>> debug is enabled.
>>
>> 2017-09-22 09:35:26,462 INFO  [stdout] (MSC service thread 1-4)   Version:
>> V3
>> 2017-09-22 09:35:26,464 INFO  [stdout] (MSC service thread 1-4)   Subject:
>> CN=engine01.mydomain.za, O=mydomain, C=US
>> 2017-09-22 09:35:26,467 INFO  [stdout] (MSC service thread 1-4)
>> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5
>> 2017-09-22 09:35:26,471 INFO  [stdout] (MSC service thread 1-4)
>> 2017-09-22 09:35:26,472 INFO  [stdout] (MSC service thread 1-4)   Key:
>> Sun RSA public key, 1024 bits
>> 2017-09-22 09:35:26,474 INFO  [stdout] (MSC service thread 1-4)   modulus:
>> 96670613185023785772001656613227416922514371649313203413281121371175732119596513752882171306045450346018887835032223373125981220753972276294203593174404470265593368091683564110524316403260121331609213962612618181708680331850541390318868926054438078223371655800890725486783860059873397983318033852172060923531
>> 2017-09-22 09:35:26,476 INFO  [stdout] (MSC service thread 1-4)   public
>> exponent: 65537
>> 2017-09-22 09:35:26,477 INFO  [stdout] (MSC service thread 1-4)
>> Validity: [From: Sun Oct 14 22:26:46 SAST 2012,
>> 2017-09-22 09:35:26,478 INFO  [stdout] (MSC service thread 1-4)
>> To: Tue Sep 19 18:26:49 SAST 2017]
>> 2017-09-22 09:35:26,479 INFO  [stdout] (MSC service thread 1-4)   Issuer:
>> CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
>>
>> Any idea how I can generate a new one and what cert it is that's expired?
>
>
> It seems that your engine certificate has expired, but AFAIK this
> certificate should be automatically renewed during engine-setup. So when did
> you execute engine-setup for last time? Any info/warning about this shown
> during invocation?

Correct, Martin was a bit faster then me :)

>
> Also looking at server.log I found JBoss 7.1.1, so you are using really
> ancient oVirt, version, right?
>
>>
>> Please see the attached log for more info.
>>
>> Thank you so much for your assistance.
>>
>> Regards.
>>
>> Neil Wilson.
>>
>>
>>
>>
>>
>>
>> On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski
>> <piotr.kliczewski@gmail.com> wrote:
>>>
>>> Neil,
>>>
>>> It seems that your engine certificate(s) is/are not ok. I would
>>> suggest to enable ssl debug in the engine by:
>>> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1].
>>> - restart your engine
>>> - check your server.log and check what is the issue.
>>>
>>> Hopefully we will be able to understand what happened in your setup.
>>>
>>> Thanks,
>>> Piotr
>>>
>>> [1]
>>> https://github.com/oVirt/ovirt-engine/blob/master/packaging/services/ovirt-engine/ovirt-engine.py#L341
>>>
>>> On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote:
>>> > Further to the logs sent, on the nodes I'm also seeing the following
>>> > error
>>> > under /var/log/messages...
>>> >
>>> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with
>>> > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C
>>> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler
>>> > exception#012Traceback
>>> > (most recent call last):#012  File "/usr/share/vdsm/BindingXMLRPC.py",
>>> > line
>>> > 80, in threaded_start#012    self.server.handle_request()#012  File
>>> > "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012
>>> > self._handle_request_noblock()#012  File
>>> > "/usr/lib64/python2.6/SocketServer.py", line 288, in
>>> > _handle_request_noblock#012    request, client_address =
>>> > self.get_request()#012  File "/usr/lib64/python2.6/SocketServer.py",
>>> > line
>>> > 456, in get_request#012    return self.socket.accept()#012  File
>>> > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line
>>> > 136,
>>> > in accept#012    raise SSL.SSLError("%s, client %s" % (e,
>>> > address[0]))#012SSLError: no certificate returned, client 10.251.193.5
>>> >
>>> > Not sure if this is any further help in diagnosing the issue?
>>> >
>>> > Thanks, any assistance is appreciated.
>>> >
>>> > Regards.
>>> >
>>> > Neil Wilson.
>>> >
>>> >
>>> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
>>> >>
>>> >> Hi Piotr,
>>> >>
>>> >> Thank you for the reply. After sending the email I did go and check
>>> >> the
>>> >> engine one too....
>>> >>
>>> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem
>>> >> -enddate
>>> >> -noout
>>> >> notAfter=Oct 13 16:26:46 2022 GMT
>>> >>
>>> >> I'm not sure if this one below is meant to verify or if this output is
>>> >> expected?
>>> >>
>>> >> [root@engine01 /]# openssl x509 -in
>>> >> /etc/pki/ovirt-engine/private/ca.pem
>>> >> -enddate -noout
>>> >> unable to load certificate
>>> >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start
>>> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
>>> >>
>>> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017
>>> >>
>>> >> Any ideas?
>>> >>
>>> >> Googling surprisingly doesn't come up with much.
>>> >>
>>> >> Thank you.
>>> >>
>>> >> Regards.
>>> >>
>>> >> Neil Wilson.
>>> >>
>>> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski
>>> >> <piotr.kliczewski@gmail.com> wrote:
>>> >>>
>>> >>> Neil,
>>> >>>
>>> >>> You checked both nodes what about the engine? Can you check engine
>>> >>> certs?
>>> >>> You can find more info where they are located here [1].
>>> >>>
>>> >>> Thanks,
>>> >>> Piotr
>>> >>>
>>> >>> [1]
>>> >>>
>>> >>> https://www.ovirt.org/develop/release-management/features/infra/pki/#ovirt-engine
>>> >>>
>>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
>>> >>> > Hi guys,
>>> >>> >
>>> >>> > Please could someone assist, my cluster is down and I can't access
>>> >>> > my
>>> >>> > vm's
>>> >>> > to switch some of them back on.
>>> >>> >
>>> >>> > I'm seeing the following error in the engine.log however I've
>>> >>> > checked
>>> >>> > my
>>> >>> > certs on my hosts (as some of the goolge results said to check),
>>> >>> > but
>>> >>> > the
>>> >>> > certs haven't expired...
>>> >>> >
>>> >>> >
>>> >>> > 2017-09-21 15:09:45,077 ERROR
>>> >>> >
>>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>> >>> > (DefaultQuartzScheduler_Worker-4) Command
>>> >>> > GetCapabilitiesVDSCommand(HostName
>>> >>> > = node02.mydomain.za, HostId =
>>> >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4,
>>> >>> > vds=Host[node02.mydomain.za]) execution failed. Exception:
>>> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received
>>> >>> > fatal
>>> >>> > alert: certificate_expired
>>> >>> > 2017-09-21 15:09:45,086 ERROR
>>> >>> >
>>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>> >>> > (DefaultQuartzScheduler_Worker-10) Command
>>> >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId =
>>> >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za])
>>> >>> > execution failed. Exception: VDSNetworkException:
>>> >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert:
>>> >>> > certificate_expired
>>> >>> > 2017-09-21 15:09:48,173 ERROR
>>> >>> >
>>> >>> > My engine and host info is below...
>>> >>> >
>>> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt
>>> >>> > ovirt-engine-lib-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch
>>> >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-3.4.0-1.el6.noarch
>>> >>> > ovirt-host-deploy-1.2.0-1.el6.noarch
>>> >>> > ovirt-engine-backend-3.4.0-1.el6.noarch
>>> >>> > ovirt-image-uploader-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-tools-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch
>>> >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch
>>> >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch
>>> >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch
>>> >>> > ovirt-log-collector-3.4.1-1.el6.noarch
>>> >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch
>>> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release
>>> >>> > CentOS release 6.5 (Final)
>>> >>> >
>>> >>> >
>>> >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem
>>> >>> > -enddate
>>> >>> > -noout ; date
>>> >>> > notAfter=May 27 08:36:17 2019 GMT
>>> >>> > Thu Sep 21 15:18:22 SAST 2017
>>> >>> > CentOS release 6.5 (Final)
>>> >>> > [root@node02 ~]# rpm -qa | grep vdsm
>>> >>> > vdsm-4.14.6-0.el6.x86_64
>>> >>> > vdsm-python-4.14.6-0.el6.x86_64
>>> >>> > vdsm-cli-4.14.6-0.el6.noarch
>>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch
>>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch
>>> >>> >
>>> >>> >
>>> >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem
>>> >>> > -enddate
>>> >>> > -noout ; date
>>> >>> > notAfter=Jun 13 16:09:41 2018 GMT
>>> >>> > Thu Sep 21 15:18:52 SAST 2017
>>> >>> > CentOS release 6.5 (Final)
>>> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm
>>> >>> > vdsm-4.14.6-0.el6.x86_64
>>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch
>>> >>> > vdsm-cli-4.14.6-0.el6.noarch
>>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch
>>> >>> > vdsm-python-4.14.6-0.el6.x86_64
>>> >>> >
>>> >>> > Please could I have some assistance, I'm rater desperate.
>>> >>> >
>>> >>> > Thank you.
>>> >>> >
>>> >>> > Regards.
>>> >>> >
>>> >>> > Neil Wilson
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > Users mailing list
>>> >>> > Users@ovirt.org
>>> >>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >>> >
>>> >>
>>> >>
>>> >
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>