SSLHandshakeException: Received fatal alert: certificate_expired

Hi guys, Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on. I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired... 2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR My engine and host info is below... [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final) [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64 Please could I have some assistance, I'm rater desperate. Thank you. Regards. Neil Wilson

Neil, You checked both nodes what about the engine? Can you check engine certs? You can find more info where they are located here [1]. Thanks, Piotr [1] https://www.ovirt.org/develop/release-management/features/infra/pki/#ovirt-e... On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Hi Piotr, Thank you for the reply. After sending the email I did go and check the engine one too.... [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -enddate -noout notAfter=Oct 13 16:26:46 2022 GMT I'm not sure if this one below is meant to verify or if this output is expected? [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE My date is correct too Thu Sep 21 16:30:15 SAST 2017 Any ideas? Googling surprisingly doesn't come up with much. Thank you. Regards. Neil Wilson. On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
Neil,
You checked both nodes what about the engine? Can you check engine certs? You can find more info where they are located here [1].
Thanks, Piotr
[1] https://www.ovirt.org/develop/release-management/features/ infra/pki/#ovirt-engine
On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand( HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages... Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5 Not sure if this is any further help in diagnosing the issue? Thanks, any assistance is appreciated. Regards. Neil Wilson. On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the reply. After sending the email I did go and check the engine one too....
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -enddate -noout notAfter=Oct 13 16:26:46 2022 GMT
I'm not sure if this one below is meant to verify or if this output is expected?
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
My date is correct too Thu Sep 21 16:30:15 SAST 2017
Any ideas?
Googling surprisingly doesn't come up with much.
Thank you.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
Neil,
You checked both nodes what about the engine? Can you check engine certs? You can find more info where they are located here [1].
Thanks, Piotr
[1] https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine
On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Neil, It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue. Hopefully we will be able to understand what happened in your setup. Thanks, Piotr [1] https://github.com/oVirt/ovirt-engine/blob/master/packaging/services/ovirt-e... On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote:
Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages...
Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5
Not sure if this is any further help in diagnosing the issue?
Thanks, any assistance is appreciated.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the reply. After sending the email I did go and check the engine one too....
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -enddate -noout notAfter=Oct 13 16:26:46 2022 GMT
I'm not sure if this one below is meant to verify or if this output is expected?
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
My date is correct too Thu Sep 21 16:30:15 SAST 2017
Any ideas?
Googling surprisingly doesn't come up with much.
Thank you.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
You checked both nodes what about the engine? Can you check engine certs? You can find more info where they are located here [1].
Thanks, Piotr
[1] https://www.ovirt.org/develop/release-management/features/infra/pki/#ovirt-e...
On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Hi Piotr, Thank you for the information. It looks like something has expired looking in the server.log now that debug is enabled. 2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version: V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 96670613185023785772001656613227416922514371649313203413281121371175732119596513752882171306045450346018887835032223373125981220753972276294203593174404470265593368091683564110524316403260121331609213962612618181708680331850541390318868926054438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) public exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US Any idea how I can generate a new one and what cert it is that's expired? Please see the attached log for more info. Thank you so much for your assistance. Regards. Neil Wilson. On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
Neil,
It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue.
Hopefully we will be able to understand what happened in your setup.
Thanks, Piotr
[1] https://github.com/oVirt/ovirt-engine/blob/master/ packaging/services/ovirt-engine/ovirt-engine.py#L341
Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages...
Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC.py",
80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketServer.py",
On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote: line line
456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5
Not sure if this is any further help in diagnosing the issue?
Thanks, any assistance is appreciated.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the reply. After sending the email I did go and check the engine one too....
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem
-enddate
-noout notAfter=Oct 13 16:26:46 2022 GMT
I'm not sure if this one below is meant to verify or if this output is expected?
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
My date is correct too Thu Sep 21 16:30:15 SAST 2017
Any ideas?
Googling surprisingly doesn't come up with much.
Thank you.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
You checked both nodes what about the engine? Can you check engine
certs?
You can find more info where they are located here [1].
Thanks, Piotr
[1] https://www.ovirt.org/develop/release-management/features/ infra/pki/#ovirt-engine
On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker. GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker. GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the information.
It looks like something has expired looking in the server.log now that debug is enabled.
2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version: V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) public exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation? Also looking at server.log I found JBoss 7.1.1, so you are using really ancient oVirt, version, right?
Please see the attached log for more info.
Thank you so much for your assistance.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
Neil,
It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue.
Hopefully we will be able to understand what happened in your setup.
Thanks, Piotr
[1] https://github.com/oVirt/ovirt-engine/blob/master/packaging/ services/ovirt-engine/ovirt-engine.py#L341
Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages...
Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC.py",
80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketServer.py",
On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote: line line
456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5
Not sure if this is any further help in diagnosing the issue?
Thanks, any assistance is appreciated.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the reply. After sending the email I did go and check the engine one too....
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem
-enddate
-noout notAfter=Oct 13 16:26:46 2022 GMT
I'm not sure if this one below is meant to verify or if this output is expected?
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
My date is correct too Thu Sep 21 16:30:15 SAST 2017
Any ideas?
Googling surprisingly doesn't come up with much.
Thank you.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
You checked both nodes what about the engine? Can you check engine
certs?
You can find more info where they are located here [1].
Thanks, Piotr
[1] https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine
On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f5 0f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the information.
It looks like something has expired looking in the server.log now that debug is enabled.
2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version: V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 96670613185023785772001656613227416922514371649313203413281121371175732119596513752882171306045450346018887835032223373125981220753972276294203593174404470265593368091683564110524316403260121331609213962612618181708680331850541390318868926054438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) public exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation?
Correct, Martin was a bit faster then me :)
Also looking at server.log I found JBoss 7.1.1, so you are using really ancient oVirt, version, right?
Please see the attached log for more info.
Thank you so much for your assistance.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue.
Hopefully we will be able to understand what happened in your setup.
Thanks, Piotr
[1] https://github.com/oVirt/ovirt-engine/blob/master/packaging/services/ovirt-e...
On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote:
Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages...
Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC.py", line 80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line 136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5
Not sure if this is any further help in diagnosing the issue?
Thanks, any assistance is appreciated.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the reply. After sending the email I did go and check the engine one too....
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -enddate -noout notAfter=Oct 13 16:26:46 2022 GMT
I'm not sure if this one below is meant to verify or if this output is expected?
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
My date is correct too Thu Sep 21 16:30:15 SAST 2017
Any ideas?
Googling surprisingly doesn't come up with much.
Thank you.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
You checked both nodes what about the engine? Can you check engine certs? You can find more info where they are located here [1].
Thanks, Piotr
[1]
https://www.ovirt.org/develop/release-management/features/infra/pki/#ovirt-e...
On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: > Hi guys, > > Please could someone assist, my cluster is down and I can't access > my > vm's > to switch some of them back on. > > I'm seeing the following error in the engine.log however I've > checked > my > certs on my hosts (as some of the goolge results said to check), > but > the > certs haven't expired... > > > 2017-09-21 15:09:45,077 ERROR > > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler_Worker-4) Command > GetCapabilitiesVDSCommand(HostName > = node02.mydomain.za, HostId = > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, > vds=Host[node02.mydomain.za]) execution failed. Exception: > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received > fatal > alert: certificate_expired > 2017-09-21 15:09:45,086 ERROR > > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] > (DefaultQuartzScheduler_Worker-10) Command > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) > execution failed. Exception: VDSNetworkException: > javax.net.ssl.SSLHandshakeException: Received fatal alert: > certificate_expired > 2017-09-21 15:09:48,173 ERROR > > My engine and host info is below... > > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt > ovirt-engine-lib-3.4.0-1.el6.noarch > ovirt-engine-restapi-3.4.0-1.el6.noarch > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch > ovirt-engine-3.4.0-1.el6.noarch > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch > ovirt-host-deploy-java-1.2.0-1.el6.noarch > ovirt-engine-setup-3.4.0-1.el6.noarch > ovirt-host-deploy-1.2.0-1.el6.noarch > ovirt-engine-backend-3.4.0-1.el6.noarch > ovirt-image-uploader-3.4.0-1.el6.noarch > ovirt-engine-tools-3.4.0-1.el6.noarch > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch > ovirt-engine-cli-3.4.0.5-1.el6.noarch > ovirt-engine-setup-base-3.4.0-1.el6.noarch > ovirt-iso-uploader-3.4.0-1.el6.noarch > ovirt-engine-userportal-3.4.0-1.el6.noarch > ovirt-log-collector-3.4.1-1.el6.noarch > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch > ovirt-engine-dbscripts-3.4.0-1.el6.noarch > [root@engine01 ovirt-engine]# cat /etc/redhat-release > CentOS release 6.5 (Final) > > > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem > -enddate > -noout ; date > notAfter=May 27 08:36:17 2019 GMT > Thu Sep 21 15:18:22 SAST 2017 > CentOS release 6.5 (Final) > [root@node02 ~]# rpm -qa | grep vdsm > vdsm-4.14.6-0.el6.x86_64 > vdsm-python-4.14.6-0.el6.x86_64 > vdsm-cli-4.14.6-0.el6.noarch > vdsm-xmlrpc-4.14.6-0.el6.noarch > vdsm-python-zombiereaper-4.14.6-0.el6.noarch > > > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem > -enddate > -noout ; date > notAfter=Jun 13 16:09:41 2018 GMT > Thu Sep 21 15:18:52 SAST 2017 > CentOS release 6.5 (Final) > [root@node01 ~]# rpm -qa | grep -i vdsm > vdsm-4.14.6-0.el6.x86_64 > vdsm-xmlrpc-4.14.6-0.el6.noarch > vdsm-cli-4.14.6-0.el6.noarch > vdsm-python-zombiereaper-4.14.6-0.el6.noarch > vdsm-python-4.14.6-0.el6.x86_64 > > Please could I have some assistance, I'm rater desperate. > > Thank you. > > Regards. > > Neil Wilson > > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Thanks Martin and Piotr, Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years. I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014 I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for? Thanks for the assistance. Regards. Neil Wilson On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the information.
It looks like something has expired looking in the server.log now that debug is enabled.
2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4)
Version:
V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) public exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation?
Correct, Martin was a bit faster then me :)
Also looking at server.log I found JBoss 7.1.1, so you are using really ancient oVirt, version, right?
Please see the attached log for more info.
Thank you so much for your assistance.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue.
Hopefully we will be able to understand what happened in your setup.
Thanks, Piotr
On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote:
Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages...
Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate
with
subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/ BindingXMLRPC.py", line 80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py",
136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5
Not sure if this is any further help in diagnosing the issue?
Thanks, any assistance is appreciated.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the reply. After sending the email I did go and check the engine one too....
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem -enddate -noout notAfter=Oct 13 16:26:46 2022 GMT
I'm not sure if this one below is meant to verify or if this output
is
expected?
[root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/private/ca.pem -enddate -noout unable to load certificate 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
My date is correct too Thu Sep 21 16:30:15 SAST 2017
Any ideas?
Googling surprisingly doesn't come up with much.
Thank you.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote: > > Neil, > > You checked both nodes what about the engine? Can you check engine > certs? > You can find more info where they are located here [1]. > > Thanks, > Piotr > > [1] > > https://www.ovirt.org/develop/release-management/features/ infra/pki/#ovirt-engine > > On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: > > Hi guys, > > > > Please could someone assist, my cluster is down and I can't access > > my > > vm's > > to switch some of them back on. > > > > I'm seeing the following error in the engine.log however I've > > checked > > my > > certs on my hosts (as some of the goolge results said to check), > > but > > the > > certs haven't expired... > > > > > > 2017-09-21 15:09:45,077 ERROR > > > > [org.ovirt.engine.core.vdsbroker.vdsbroker. GetCapabilitiesVDSCommand] > > (DefaultQuartzScheduler_Worker-4) Command > > GetCapabilitiesVDSCommand(HostName > > = node02.mydomain.za, HostId = > > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, > > vds=Host[node02.mydomain.za]) execution failed. Exception: > > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received > > fatal > > alert: certificate_expired > > 2017-09-21 15:09:45,086 ERROR > > > > [org.ovirt.engine.core.vdsbroker.vdsbroker. GetCapabilitiesVDSCommand] > > (DefaultQuartzScheduler_Worker-10) Command > > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = > > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ node01.mydomain.za]) > > execution failed. Exception: VDSNetworkException: > > javax.net.ssl.SSLHandshakeException: Received fatal alert: > > certificate_expired > > 2017-09-21 15:09:48,173 ERROR > > > > My engine and host info is below... > > > > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt > > ovirt-engine-lib-3.4.0-1.el6.noarch > > ovirt-engine-restapi-3.4.0-1.el6.noarch > > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch > > ovirt-engine-3.4.0-1.el6.noarch > > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch > > ovirt-host-deploy-java-1.2.0-1.el6.noarch > > ovirt-engine-setup-3.4.0-1.el6.noarch > > ovirt-host-deploy-1.2.0-1.el6.noarch > > ovirt-engine-backend-3.4.0-1.el6.noarch > > ovirt-image-uploader-3.4.0-1.el6.noarch > > ovirt-engine-tools-3.4.0-1.el6.noarch > > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch > > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch > > ovirt-engine-cli-3.4.0.5-1.el6.noarch > > ovirt-engine-setup-base-3.4.0-1.el6.noarch > > ovirt-iso-uploader-3.4.0-1.el6.noarch > > ovirt-engine-userportal-3.4.0-1.el6.noarch > > ovirt-log-collector-3.4.1-1.el6.noarch > > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch > > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch > > ovirt-engine-dbscripts-3.4.0-1.el6.noarch > > [root@engine01 ovirt-engine]# cat /etc/redhat-release > > CentOS release 6.5 (Final) > > > > > > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.
> > -enddate > > -noout ; date > > notAfter=May 27 08:36:17 2019 GMT > > Thu Sep 21 15:18:22 SAST 2017 > > CentOS release 6.5 (Final) > > [root@node02 ~]# rpm -qa | grep vdsm > > vdsm-4.14.6-0.el6.x86_64 > > vdsm-python-4.14.6-0.el6.x86_64 > > vdsm-cli-4.14.6-0.el6.noarch > > vdsm-xmlrpc-4.14.6-0.el6.noarch > > vdsm-python-zombiereaper-4.14.6-0.el6.noarch > > > > > > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.
packaging/services/ovirt-engine/ovirt-engine.py#L341 line pem pem
> > -enddate > > -noout ; date > > notAfter=Jun 13 16:09:41 2018 GMT > > Thu Sep 21 15:18:52 SAST 2017 > > CentOS release 6.5 (Final) > > [root@node01 ~]# rpm -qa | grep -i vdsm > > vdsm-4.14.6-0.el6.x86_64 > > vdsm-xmlrpc-4.14.6-0.el6.noarch > > vdsm-cli-4.14.6-0.el6.noarch > > vdsm-python-zombiereaper-4.14.6-0.el6.noarch > > vdsm-python-4.14.6-0.el6.x86_64 > > > > Please could I have some assistance, I'm rater desperate. > > > > Thank you. > > > > Regards. > > > > Neil Wilson > > > > > > > > _______________________________________________ > > Users mailing list > > Users@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123@gmail.com> wrote:
Thanks Martin and Piotr,
Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.
I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014
I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?
Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either: 1. Execute engine-setup --offline to skip updates check or 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org
Thanks for the assistance.
Regards.
Neil Wilson
On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the information.
It looks like something has expired looking in the server.log now that debug is enabled.
2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4)
V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4)
Version: public
exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation?
Correct, Martin was a bit faster then me :)
Also looking at server.log I found JBoss 7.1.1, so you are using really ancient oVirt, version, right?
Please see the attached log for more info.
Thank you so much for your assistance.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue.
Hopefully we will be able to understand what happened in your setup.
Thanks, Piotr
[1] https://github.com/oVirt/ovirt-engine/blob/master/packaging/
On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote:
Further to the logs sent, on the nodes I'm also seeing the following error under /var/log/messages...
Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate
with
subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler exception#012Traceback (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC .py", line 80, in threaded_start#012 self.server.handle_request()#012 File "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 self._handle_request_noblock()#012 File "/usr/lib64/python2.6/SocketServer.py", line 288, in _handle_request_noblock#012 request, client_address = self.get_request()#012 File "/usr/lib64/python2.6/SocketSe rver.py", line 456, in get_request#012 return self.socket.accept()#012 File "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py",
services/ovirt-engine/ovirt-engine.py#L341 line
136, in accept#012 raise SSL.SSLError("%s, client %s" % (e, address[0]))#012SSLError: no certificate returned, client 10.251.193.5
Not sure if this is any further help in diagnosing the issue?
Thanks, any assistance is appreciated.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote: > > Hi Piotr, > > Thank you for the reply. After sending the email I did go and check > the > engine one too.... > > [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem > -enddate > -noout > notAfter=Oct 13 16:26:46 2022 GMT > > I'm not sure if this one below is meant to verify or if this output is > expected? > > [root@engine01 /]# openssl x509 -in > /etc/pki/ovirt-engine/private/ca.pem > -enddate -noout > unable to load certificate > 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start > line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE > > My date is correct too Thu Sep 21 16:30:15 SAST 2017 > > Any ideas? > > Googling surprisingly doesn't come up with much. > > Thank you. > > Regards. > > Neil Wilson. > > On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski > <piotr.kliczewski@gmail.com> wrote: >> >> Neil, >> >> You checked both nodes what about the engine? Can you check engine >> certs? >> You can find more info where they are located here [1]. >> >> Thanks, >> Piotr >> >> [1] >> >> https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine >> >> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: >> > Hi guys, >> > >> > Please could someone assist, my cluster is down and I can't access >> > my >> > vm's >> > to switch some of them back on. >> > >> > I'm seeing the following error in the engine.log however I've >> > checked >> > my >> > certs on my hosts (as some of the goolge results said to check), >> > but >> > the >> > certs haven't expired... >> > >> > >> > 2017-09-21 15:09:45,077 ERROR >> > >> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] >> > (DefaultQuartzScheduler_Worker-4) Command >> > GetCapabilitiesVDSCommand(HostName >> > = node02.mydomain.za, HostId = >> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, >> > vds=Host[node02.mydomain.za]) execution failed. Exception: >> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received >> > fatal >> > alert: certificate_expired >> > 2017-09-21 15:09:45,086 ERROR >> > >> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] >> > (DefaultQuartzScheduler_Worker-10) Command >> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = >> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ node01.mydomain.za]) >> > execution failed. Exception: VDSNetworkException: >> > javax.net.ssl.SSLHandshakeException: Received fatal alert: >> > certificate_expired >> > 2017-09-21 15:09:48,173 ERROR >> > >> > My engine and host info is below... >> > >> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt >> > ovirt-engine-lib-3.4.0-1.el6.noarch >> > ovirt-engine-restapi-3.4.0-1.el6.noarch >> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch >> > ovirt-engine-3.4.0-1.el6.noarch >> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch >> > ovirt-host-deploy-java-1.2.0-1.el6.noarch >> > ovirt-engine-setup-3.4.0-1.el6.noarch >> > ovirt-host-deploy-1.2.0-1.el6.noarch >> > ovirt-engine-backend-3.4.0-1.el6.noarch >> > ovirt-image-uploader-3.4.0-1.el6.noarch >> > ovirt-engine-tools-3.4.0-1.el6.noarch >> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch >> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch >> > ovirt-engine-cli-3.4.0.5-1.el6.noarch >> > ovirt-engine-setup-base-3.4.0-1.el6.noarch >> > ovirt-iso-uploader-3.4.0-1.el6.noarch >> > ovirt-engine-userportal-3.4.0-1.el6.noarch >> > ovirt-log-collector-3.4.1-1.el6.noarch >> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch >> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6. noarch >> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch >> > [root@engine01 ovirt-engine]# cat /etc/redhat-release >> > CentOS release 6.5 (Final) >> > >> > >> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >> > -enddate >> > -noout ; date >> > notAfter=May 27 08:36:17 2019 GMT >> > Thu Sep 21 15:18:22 SAST 2017 >> > CentOS release 6.5 (Final) >> > [root@node02 ~]# rpm -qa | grep vdsm >> > vdsm-4.14.6-0.el6.x86_64 >> > vdsm-python-4.14.6-0.el6.x86_64 >> > vdsm-cli-4.14.6-0.el6.noarch >> > vdsm-xmlrpc-4.14.6-0.el6.noarch >> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >> > >> > >> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >> > -enddate >> > -noout ; date >> > notAfter=Jun 13 16:09:41 2018 GMT >> > Thu Sep 21 15:18:52 SAST 2017 >> > CentOS release 6.5 (Final) >> > [root@node01 ~]# rpm -qa | grep -i vdsm >> > vdsm-4.14.6-0.el6.x86_64 >> > vdsm-xmlrpc-4.14.6-0.el6.noarch >> > vdsm-cli-4.14.6-0.el6.noarch >> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >> > vdsm-python-4.14.6-0.el6.x86_64 >> > >> > Please could I have some assistance, I'm rater desperate. >> > >> > Thank you. >> > >> > Regards. >> > >> > Neil Wilson >> > >> > >> > >> > _______________________________________________ >> > Users mailing list >> > Users@ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> > > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

Thanks for the guidance everyone. I've upgraded my engine now to ovirt-engine-3.4.4-1 but I've still got the same error unfortunately. Below is the output of the upgrade. Should this have fixed the issue or do I need to upgrade to 3.5 etc? [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20170922125526-vw5khx.log Version: otopi-1.2.3 (otopi-1.2.3-1.el6) [ INFO ] Stage: Environment packages setup [ INFO ] Yum Downloading: repomdPLa0LXtmp.xml (0%) [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- --== PACKAGES ==-- [ INFO ] Checking for product updates... Setup has found updates for some packages, do you wish to update them now? (Yes, No) [Yes]: [ INFO ] Checking for an update for Setup... --== NETWORK CONFIGURATION ==-- [WARNING] Failed to resolve engine01.mydomain.za using DNS, it can be resolved only locally Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. Do you want Setup to configure the firewall? (Yes, No) [Yes]: no --== DATABASE CONFIGURATION ==-- --== OVIRT ENGINE CONFIGURATION ==-- Skipping storing options as database already prepared --== PKI CONFIGURATION ==-- PKI is already configured --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [WARNING] Less than 16384MB of memory is available [ INFO ] Cleaning stale zombie tasks --== CONFIGURATION PREVIEW ==-- Engine database name : engine Engine database secured connection : False Engine database host : localhost Engine database user name : engine Engine database host name validation : False Engine database port : 5432 Datacenter storage type : False Update Firewall : False Configure WebSocket Proxy : True Host FQDN : engine01.mydomain.za Upgrade packages : True Please confirm installation settings (OK, Cancel) [OK]: [ INFO ] Cleaning async tasks and compensations [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Yum Status: Downloading Packages [ INFO ] Yum Download/Verify: ovirt-engine-3.4.4-1.el6.noarch [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 2.0 M(19%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 4.3 M(41%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 6.3 M(60%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 8.9 M(85%) [ INFO ] Yum Download/Verify: ovirt-engine-backend-3.4.4-1.el6.noarch [ INFO ] Yum Download/Verify: ovirt-engine-dbscripts-3.4.4-1.el6.noarch (I've taken out all the downloading progress) [ INFO ] Yum Verify: 26/26: ovirt-engine-backend.noarch 0:3.4.0-1.el6 - ud [ INFO ] Stage: Misc configuration [ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20170922143709.m_8fr_.dump'. [ INFO ] Updating Engine database schema [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up --== SUMMARY ==-- [WARNING] Less than 16384MB of memory is available SSH fingerprint: 86:C7:AA:35:45:E9:83:3E:16:C9:2A:F5:68:52:68:84 Internal CA EE:91:B3:E7:40:D7:DD:A7:DD:77:9C:3B:D5:A1:E7:BE:E2:C9:8B:AA Web access is enabled at: http://engine01.mydomain.za:80/ovirt-engine https://engine01.mydomain.za:443/ovirt-engine In order to configure firewalld, copy the files from /etc/ovirt-engine/firewalld to /etc/firewalld/services and execute the following commands: firewall-cmd -service ovirt-postgres firewall-cmd -service ovirt-https firewall-cmd -service ovirt-websocket-proxy firewall-cmd -service ovirt-http The following network ports should be opened: tcp:443 tcp:5432 tcp:6100 tcp:80 An example of the required configuration for iptables can be found at: /etc/ovirt-engine/iptables.example --== END OF SUMMARY ==-- [ INFO ] Starting engine service [ INFO ] Restarting httpd [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20170922125526-vw5khx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20170922143806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully I'm still seeing the following below, in my engine.log and when I log in, all my VM's show as unknown. 2017-09-22 15:06:06,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za,d2debdfe-76e7-40cf-a7fd-78a0f50f14d4]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired Any ideas? Thanks! On Fri, Sep 22, 2017 at 11:10 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123@gmail.com> wrote:
Thanks Martin and Piotr,
Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.
I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014
I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?
Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either:
1. Execute engine-setup --offline to skip updates check or 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org
Thanks for the assistance.
Regards.
Neil Wilson
On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the information.
It looks like something has expired looking in the server.log now that debug is enabled.
2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4)
V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4)
Version: public
exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation?
Correct, Martin was a bit faster then me :)
Also looking at server.log I found JBoss 7.1.1, so you are using really ancient oVirt, version, right?
Please see the attached log for more info.
Thank you so much for your assistance.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote:
Neil,
It seems that your engine certificate(s) is/are not ok. I would suggest to enable ssl debug in the engine by: - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. - restart your engine - check your server.log and check what is the issue.
Hopefully we will be able to understand what happened in your setup.
Thanks, Piotr
[1] https://github.com/oVirt/ovirt-engine/blob/master/packaging/
On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote: > Further to the logs sent, on the nodes I'm also seeing the
following
> error > under /var/log/messages... > > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler > exception#012Traceback > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC .py", > line > 80, in threaded_start#012 self.server.handle_request()#012 File > "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 > self._handle_request_noblock()#012 File > "/usr/lib64/python2.6/SocketServer.py", line 288, in > _handle_request_noblock#012 request, client_address = > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe rver.py", > line > 456, in get_request#012 return self.socket.accept()#012 File > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py",
services/ovirt-engine/ovirt-engine.py#L341 line
> 136, > in accept#012 raise SSL.SSLError("%s, client %s" % (e, > address[0]))#012SSLError: no certificate returned, client 10.251.193.5 > > Not sure if this is any further help in diagnosing the issue? > > Thanks, any assistance is appreciated. > > Regards. > > Neil Wilson. > > > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote: >> >> Hi Piotr, >> >> Thank you for the reply. After sending the email I did go and check >> the >> engine one too.... >> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem >> -enddate >> -noout >> notAfter=Oct 13 16:26:46 2022 GMT >> >> I'm not sure if this one below is meant to verify or if this output is >> expected? >> >> [root@engine01 /]# openssl x509 -in >> /etc/pki/ovirt-engine/private/ca.pem >> -enddate -noout >> unable to load certificate >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE >> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017 >> >> Any ideas? >> >> Googling surprisingly doesn't come up with much. >> >> Thank you. >> >> Regards. >> >> Neil Wilson. >> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski >> <piotr.kliczewski@gmail.com> wrote: >>> >>> Neil, >>> >>> You checked both nodes what about the engine? Can you check engine >>> certs? >>> You can find more info where they are located here [1]. >>> >>> Thanks, >>> Piotr >>> >>> [1] >>> >>> https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine >>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: >>> > Hi guys, >>> > >>> > Please could someone assist, my cluster is down and I can't access >>> > my >>> > vm's >>> > to switch some of them back on. >>> > >>> > I'm seeing the following error in the engine.log however I've >>> > checked >>> > my >>> > certs on my hosts (as some of the goolge results said to check), >>> > but >>> > the >>> > certs haven't expired... >>> > >>> > >>> > 2017-09-21 15:09:45,077 ERROR >>> > >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] >>> > (DefaultQuartzScheduler_Worker-4) Command >>> > GetCapabilitiesVDSCommand(HostName >>> > = node02.mydomain.za, HostId = >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, >>> > vds=Host[node02.mydomain.za]) execution failed. Exception: >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received >>> > fatal >>> > alert: certificate_expired >>> > 2017-09-21 15:09:45,086 ERROR >>> > >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] >>> > (DefaultQuartzScheduler_Worker-10) Command >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ node01.mydomain.za]) >>> > execution failed. Exception: VDSNetworkException: >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert: >>> > certificate_expired >>> > 2017-09-21 15:09:48,173 ERROR >>> > >>> > My engine and host info is below... >>> > >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt >>> > ovirt-engine-lib-3.4.0-1.el6.noarch >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch >>> > ovirt-engine-3.4.0-1.el6.noarch >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch >>> > ovirt-engine-setup-3.4.0-1.el6.noarch >>> > ovirt-host-deploy-1.2.0-1.el6.noarch >>> > ovirt-engine-backend-3.4.0-1.el6.noarch >>> > ovirt-image-uploader-3.4.0-1.el6.noarch >>> > ovirt-engine-tools-3.4.0-1.el6.noarch >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch >>> > ovirt-log-collector-3.4.1-1.el6.noarch >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.no arch >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release >>> > CentOS release 6.5 (Final) >>> > >>> > >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >>> > -enddate >>> > -noout ; date >>> > notAfter=May 27 08:36:17 2019 GMT >>> > Thu Sep 21 15:18:22 SAST 2017 >>> > CentOS release 6.5 (Final) >>> > [root@node02 ~]# rpm -qa | grep vdsm >>> > vdsm-4.14.6-0.el6.x86_64 >>> > vdsm-python-4.14.6-0.el6.x86_64 >>> > vdsm-cli-4.14.6-0.el6.noarch >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >>> > >>> > >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >>> > -enddate >>> > -noout ; date >>> > notAfter=Jun 13 16:09:41 2018 GMT >>> > Thu Sep 21 15:18:52 SAST 2017 >>> > CentOS release 6.5 (Final) >>> > [root@node01 ~]# rpm -qa | grep -i vdsm >>> > vdsm-4.14.6-0.el6.x86_64 >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >>> > vdsm-cli-4.14.6-0.el6.noarch >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >>> > vdsm-python-4.14.6-0.el6.x86_64 >>> > >>> > Please could I have some assistance, I'm rater desperate. >>> > >>> > Thank you. >>> > >>> > Regards. >>> > >>> > Neil Wilson >>> > >>> > >>> > >>> > _______________________________________________ >>> > Users mailing list >>> > Users@ovirt.org >>> > http://lists.ovirt.org/mailman/listinfo/users >>> > >> >> >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

2017-09-22 15:07 GMT+02:00 Neil <nwilson123@gmail.com>:
Thanks for the guidance everyone.
I've upgraded my engine now to ovirt-engine-3.4.4-1 but I've still got the same error unfortunately. Below is the output of the upgrade. Should this have fixed the issue or do I need to upgrade to 3.5 etc?
I think you'll need 3.5.4 at least: https://bugzilla.redhat.com/show_bug.cgi?id=1214860
[ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup- 20170922125526-vw5khx.log Version: otopi-1.2.3 (otopi-1.2.3-1.el6) [ INFO ] Stage: Environment packages setup [ INFO ] Yum Downloading: repomdPLa0LXtmp.xml (0%) [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization
--== PRODUCT OPTIONS ==--
--== PACKAGES ==--
[ INFO ] Checking for product updates... Setup has found updates for some packages, do you wish to update them now? (Yes, No) [Yes]: [ INFO ] Checking for an update for Setup...
--== NETWORK CONFIGURATION ==--
[WARNING] Failed to resolve engine01.mydomain.za using DNS, it can be resolved only locally Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. Do you want Setup to configure the firewall? (Yes, No) [Yes]: no
--== DATABASE CONFIGURATION ==--
--== OVIRT ENGINE CONFIGURATION ==--
Skipping storing options as database already prepared
--== PKI CONFIGURATION ==--
PKI is already configured
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [WARNING] Less than 16384MB of memory is available [ INFO ] Cleaning stale zombie tasks
--== CONFIGURATION PREVIEW ==--
Engine database name : engine Engine database secured connection : False Engine database host : localhost Engine database user name : engine Engine database host name validation : False Engine database port : 5432 Datacenter storage type : False Update Firewall : False Configure WebSocket Proxy : True Host FQDN : engine01.mydomain.za Upgrade packages : True
Please confirm installation settings (OK, Cancel) [OK]: [ INFO ] Cleaning async tasks and compensations [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Yum Status: Downloading Packages [ INFO ] Yum Download/Verify: ovirt-engine-3.4.4-1.el6.noarch [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 2.0 M(19%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 4.3 M(41%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 6.3 M(60%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 8.9 M(85%) [ INFO ] Yum Download/Verify: ovirt-engine-backend-3.4.4-1.el6.noarch [ INFO ] Yum Download/Verify: ovirt-engine-dbscripts-3.4.4-1.el6.noarch (I've taken out all the downloading progress)
[ INFO ] Yum Verify: 26/26: ovirt-engine-backend.noarch 0:3.4.0-1.el6 - ud [ INFO ] Stage: Misc configuration [ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/ backups/engine-20170922143709.m_8fr_.dump'. [ INFO ] Updating Engine database schema [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up
--== SUMMARY ==--
[WARNING] Less than 16384MB of memory is available SSH fingerprint: 86:C7:AA:35:45:E9:83:3E:16:C9:2A:F5:68:52:68:84 Internal CA EE:91:B3:E7:40:D7:DD:A7:DD:77: 9C:3B:D5:A1:E7:BE:E2:C9:8B:AA Web access is enabled at: http://engine01.mydomain.za:80/ovirt-engine https://engine01.mydomain.za:443/ovirt-engine In order to configure firewalld, copy the files from /etc/ovirt-engine/firewalld to /etc/firewalld/services and execute the following commands: firewall-cmd -service ovirt-postgres firewall-cmd -service ovirt-https firewall-cmd -service ovirt-websocket-proxy firewall-cmd -service ovirt-http The following network ports should be opened: tcp:443 tcp:5432 tcp:6100 tcp:80 An example of the required configuration for iptables can be found at: /etc/ovirt-engine/iptables.example
--== END OF SUMMARY ==--
[ INFO ] Starting engine service [ INFO ] Restarting httpd [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ ovirt-engine-setup-20170922125526-vw5khx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/ answers/20170922143806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully
I'm still seeing the following below, in my engine.log and when I log in, all my VM's show as unknown.
2017-09-22 15:06:06,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za,d2debdfe-76e7-40cf-a7fd-78a0f50f14d4]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired
Any ideas?
Thanks!
On Fri, Sep 22, 2017 at 11:10 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123@gmail.com> wrote:
Thanks Martin and Piotr,
Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.
I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014
I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?
Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either:
1. Execute engine-setup --offline to skip updates check or 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org
Thanks for the assistance.
Regards.
Neil Wilson
On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
Hi Piotr,
Thank you for the information.
It looks like something has expired looking in the server.log now
debug is enabled.
2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version: V3 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: CN=engine01.mydomain.za, O=mydomain, C=US 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: Sun RSA public key, 1024 bits 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4)
that public
exponent: 65537 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) Validity: [From: Sun Oct 14 22:26:46 SAST 2012, 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) To: Tue Sep 19 18:26:49 SAST 2017] 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation?
Correct, Martin was a bit faster then me :)
Also looking at server.log I found JBoss 7.1.1, so you are using
ancient oVirt, version, right?
Please see the attached log for more info.
Thank you so much for your assistance.
Regards.
Neil Wilson.
On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski <piotr.kliczewski@gmail.com> wrote: > > Neil, > > It seems that your engine certificate(s) is/are not ok. I would > suggest to enable ssl debug in the engine by: > - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. > - restart your engine > - check your server.log and check what is the issue. > > Hopefully we will be able to understand what happened in your setup. > > Thanks, > Piotr > > [1] > https://github.com/oVirt/ovirt-engine/blob/master/packaging/
services/ovirt-engine/ovirt-engine.py#L341
> > On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote: > > Further to the logs sent, on the nodes I'm also seeing the following > > error > > under /var/log/messages... > > > > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with > > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C > > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler > > exception#012Traceback > > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC .py", > > line > > 80, in threaded_start#012 self.server.handle_request()#012 File > > "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 > > self._handle_request_noblock()#012 File > > "/usr/lib64/python2.6/SocketServer.py", line 288, in > > _handle_request_noblock#012 request, client_address = > > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe rver.py", > > line > > 456, in get_request#012 return self.socket.accept()#012 File > > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py",
really line
> > 136, > > in accept#012 raise SSL.SSLError("%s, client %s" % (e, > > address[0]))#012SSLError: no certificate returned, client 10.251.193.5 > > > > Not sure if this is any further help in diagnosing the issue? > > > > Thanks, any assistance is appreciated. > > > > Regards. > > > > Neil Wilson. > > > > > > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote: > >> > >> Hi Piotr, > >> > >> Thank you for the reply. After sending the email I did go and check > >> the > >> engine one too.... > >> > >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem > >> -enddate > >> -noout > >> notAfter=Oct 13 16:26:46 2022 GMT > >> > >> I'm not sure if this one below is meant to verify or if this output is > >> expected? > >> > >> [root@engine01 /]# openssl x509 -in > >> /etc/pki/ovirt-engine/private/ca.pem > >> -enddate -noout > >> unable to load certificate > >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start > >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE > >> > >> My date is correct too Thu Sep 21 16:30:15 SAST 2017 > >> > >> Any ideas? > >> > >> Googling surprisingly doesn't come up with much. > >> > >> Thank you. > >> > >> Regards. > >> > >> Neil Wilson. > >> > >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski > >> <piotr.kliczewski@gmail.com> wrote: > >>> > >>> Neil, > >>> > >>> You checked both nodes what about the engine? Can you check engine > >>> certs? > >>> You can find more info where they are located here [1]. > >>> > >>> Thanks, > >>> Piotr > >>> > >>> [1] > >>> > >>> https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine > >>> > >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: > >>> > Hi guys, > >>> > > >>> > Please could someone assist, my cluster is down and I can't access > >>> > my > >>> > vm's > >>> > to switch some of them back on. > >>> > > >>> > I'm seeing the following error in the engine.log however I've > >>> > checked > >>> > my > >>> > certs on my hosts (as some of the goolge results said to check), > >>> > but > >>> > the > >>> > certs haven't expired... > >>> > > >>> > > >>> > 2017-09-21 15:09:45,077 ERROR > >>> > > >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] > >>> > (DefaultQuartzScheduler_Worker-4) Command > >>> > GetCapabilitiesVDSCommand(HostName > >>> > = node02.mydomain.za, HostId = > >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, > >>> > vds=Host[node02.mydomain.za]) execution failed. Exception: > >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received > >>> > fatal > >>> > alert: certificate_expired > >>> > 2017-09-21 15:09:45,086 ERROR > >>> > > >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] > >>> > (DefaultQuartzScheduler_Worker-10) Command > >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = > >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ node01.mydomain.za]) > >>> > execution failed. Exception: VDSNetworkException: > >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert: > >>> > certificate_expired > >>> > 2017-09-21 15:09:48,173 ERROR > >>> > > >>> > My engine and host info is below... > >>> > > >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt > >>> > ovirt-engine-lib-3.4.0-1.el6.noarch > >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch > >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch > >>> > ovirt-engine-3.4.0-1.el6.noarch > >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch > >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch > >>> > ovirt-engine-setup-3.4.0-1.el6.noarch > >>> > ovirt-host-deploy-1.2.0-1.el6.noarch > >>> > ovirt-engine-backend-3.4.0-1.el6.noarch > >>> > ovirt-image-uploader-3.4.0-1.el6.noarch > >>> > ovirt-engine-tools-3.4.0-1.el6.noarch > >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch > >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch > >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch > >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch > >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch > >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch > >>> > ovirt-log-collector-3.4.1-1.el6.noarch > >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch > >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.no arch > >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch > >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release > >>> > CentOS release 6.5 (Final) > >>> > > >>> > > >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem > >>> > -enddate > >>> > -noout ; date > >>> > notAfter=May 27 08:36:17 2019 GMT > >>> > Thu Sep 21 15:18:22 SAST 2017 > >>> > CentOS release 6.5 (Final) > >>> > [root@node02 ~]# rpm -qa | grep vdsm > >>> > vdsm-4.14.6-0.el6.x86_64 > >>> > vdsm-python-4.14.6-0.el6.x86_64 > >>> > vdsm-cli-4.14.6-0.el6.noarch > >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch > >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch > >>> > > >>> > > >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem > >>> > -enddate > >>> > -noout ; date > >>> > notAfter=Jun 13 16:09:41 2018 GMT > >>> > Thu Sep 21 15:18:52 SAST 2017 > >>> > CentOS release 6.5 (Final) > >>> > [root@node01 ~]# rpm -qa | grep -i vdsm > >>> > vdsm-4.14.6-0.el6.x86_64 > >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch > >>> > vdsm-cli-4.14.6-0.el6.noarch > >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch > >>> > vdsm-python-4.14.6-0.el6.x86_64 > >>> > > >>> > Please could I have some assistance, I'm rater desperate. > >>> > > >>> > Thank you. > >>> > > >>> > Regards. > >>> > > >>> > Neil Wilson > >>> > > >>> > > >>> > > >>> > _______________________________________________ > >>> > Users mailing list > >>> > Users@ovirt.org > >>> > http://lists.ovirt.org/mailman/listinfo/users > >>> > > >> > >> > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

Thanks Sandro. I'll get cracking and report back if it fixed it. Thanks for all the help everyone. On Fri, Sep 22, 2017 at 3:14 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2017-09-22 15:07 GMT+02:00 Neil <nwilson123@gmail.com>:
Thanks for the guidance everyone.
I've upgraded my engine now to ovirt-engine-3.4.4-1 but I've still got the same error unfortunately. Below is the output of the upgrade. Should this have fixed the issue or do I need to upgrade to 3.5 etc?
I think you'll need 3.5.4 at least: https://bugzilla. redhat.com/show_bug.cgi?id=1214860
[ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ov irt-engine-setup-20170922125526-vw5khx.log Version: otopi-1.2.3 (otopi-1.2.3-1.el6) [ INFO ] Stage: Environment packages setup [ INFO ] Yum Downloading: repomdPLa0LXtmp.xml (0%) [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization
--== PRODUCT OPTIONS ==--
--== PACKAGES ==--
[ INFO ] Checking for product updates... Setup has found updates for some packages, do you wish to update them now? (Yes, No) [Yes]: [ INFO ] Checking for an update for Setup...
--== NETWORK CONFIGURATION ==--
[WARNING] Failed to resolve engine01.mydomain.za using DNS, it can be resolved only locally Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. Do you want Setup to configure the firewall? (Yes, No) [Yes]: no
--== DATABASE CONFIGURATION ==--
--== OVIRT ENGINE CONFIGURATION ==--
Skipping storing options as database already prepared
--== PKI CONFIGURATION ==--
PKI is already configured
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [WARNING] Less than 16384MB of memory is available [ INFO ] Cleaning stale zombie tasks
--== CONFIGURATION PREVIEW ==--
Engine database name : engine Engine database secured connection : False Engine database host : localhost Engine database user name : engine Engine database host name validation : False Engine database port : 5432 Datacenter storage type : False Update Firewall : False Configure WebSocket Proxy : True Host FQDN : engine01.mydomain.za Upgrade packages : True
Please confirm installation settings (OK, Cancel) [OK]: [ INFO ] Cleaning async tasks and compensations [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Yum Status: Downloading Packages [ INFO ] Yum Download/Verify: ovirt-engine-3.4.4-1.el6.noarch [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 2.0 M(19%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 4.3 M(41%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 6.3 M(60%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 8.9 M(85%) [ INFO ] Yum Download/Verify: ovirt-engine-backend-3.4.4-1.el6.noarch [ INFO ] Yum Download/Verify: ovirt-engine-dbscripts-3.4.4-1.el6.noarch (I've taken out all the downloading progress)
[ INFO ] Yum Verify: 26/26: ovirt-engine-backend.noarch 0:3.4.0-1.el6 - ud [ INFO ] Stage: Misc configuration [ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20170922143709.m_8fr_.dump'. [ INFO ] Updating Engine database schema [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up
--== SUMMARY ==--
[WARNING] Less than 16384MB of memory is available SSH fingerprint: 86:C7:AA:35:45:E9:83:3E:16:C9: 2A:F5:68:52:68:84 Internal CA EE:91:B3:E7:40:D7:DD:A7:DD:77: 9C:3B:D5:A1:E7:BE:E2:C9:8B:AA Web access is enabled at: http://engine01.mydomain.za:80/ovirt-engine https://engine01.mydomain.za:443/ovirt-engine In order to configure firewalld, copy the files from /etc/ovirt-engine/firewalld to /etc/firewalld/services and execute the following commands: firewall-cmd -service ovirt-postgres firewall-cmd -service ovirt-https firewall-cmd -service ovirt-websocket-proxy firewall-cmd -service ovirt-http The following network ports should be opened: tcp:443 tcp:5432 tcp:6100 tcp:80 An example of the required configuration for iptables can be found at: /etc/ovirt-engine/iptables.example
--== END OF SUMMARY ==--
[ INFO ] Starting engine service [ INFO ] Restarting httpd [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ov irt-engine-setup-20170922125526-vw5khx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/ 20170922143806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully
I'm still seeing the following below, in my engine.log and when I log in, all my VM's show as unknown.
2017-09-22 15:06:06,060 ERROR [org.ovirt.engine.core.vdsbrok er.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za,d2 debdfe-76e7-40cf-a7fd-78a0f50f14d4]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired
Any ideas?
Thanks!
On Fri, Sep 22, 2017 at 11:10 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123@gmail.com> wrote:
Thanks Martin and Piotr,
Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.
I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014
I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?
Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either:
1. Execute engine-setup --offline to skip updates check or 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org
Thanks for the assistance.
Regards.
Neil Wilson
On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote: > > Hi Piotr, > > Thank you for the information. > > It looks like something has expired looking in the server.log now
> debug is enabled. > > 2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version: > V3 > 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: > CN=engine01.mydomain.za, O=mydomain, C=US > 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) > Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 > 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) > 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: > Sun RSA public key, 1024 bits > 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: > 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 > 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4)
that public
> exponent: 65537 > 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) > Validity: [From: Sun Oct 14 22:26:46 SAST 2012, > 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) > To: Tue Sep 19 18:26:49 SAST 2017] > 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: > CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US > > Any idea how I can generate a new one and what cert it is that's expired?
It seems that your engine certificate has expired, but AFAIK this certificate should be automatically renewed during engine-setup. So when did you execute engine-setup for last time? Any info/warning about this shown during invocation?
Correct, Martin was a bit faster then me :)
Also looking at server.log I found JBoss 7.1.1, so you are using
ancient oVirt, version, right?
> > Please see the attached log for more info. > > Thank you so much for your assistance. > > Regards. > > Neil Wilson. > > > > > > > On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski > <piotr.kliczewski@gmail.com> wrote: >> >> Neil, >> >> It seems that your engine certificate(s) is/are not ok. I would >> suggest to enable ssl debug in the engine by: >> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. >> - restart your engine >> - check your server.log and check what is the issue. >> >> Hopefully we will be able to understand what happened in your setup. >> >> Thanks, >> Piotr >> >> [1] >> https://github.com/oVirt/ovirt-engine/blob/master/packaging/ services/ovirt-engine/ovirt-engine.py#L341 >> >> On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote: >> > Further to the logs sent, on the nodes I'm also seeing the following >> > error >> > under /var/log/messages... >> > >> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with >> > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C >> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler >> > exception#012Traceback >> > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC .py", >> > line >> > 80, in threaded_start#012 self.server.handle_request()#012 File >> > "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 >> > self._handle_request_noblock()#012 File >> > "/usr/lib64/python2.6/SocketServer.py", line 288, in >> > _handle_request_noblock#012 request, client_address = >> > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe rver.py", >> > line >> > 456, in get_request#012 return self.socket.accept()#012 File >> > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py",
really line
>> > 136, >> > in accept#012 raise SSL.SSLError("%s, client %s" % (e, >> > address[0]))#012SSLError: no certificate returned, client 10.251.193.5 >> > >> > Not sure if this is any further help in diagnosing the issue? >> > >> > Thanks, any assistance is appreciated. >> > >> > Regards. >> > >> > Neil Wilson. >> > >> > >> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote: >> >> >> >> Hi Piotr, >> >> >> >> Thank you for the reply. After sending the email I did go and check >> >> the >> >> engine one too.... >> >> >> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem >> >> -enddate >> >> -noout >> >> notAfter=Oct 13 16:26:46 2022 GMT >> >> >> >> I'm not sure if this one below is meant to verify or if this output is >> >> expected? >> >> >> >> [root@engine01 /]# openssl x509 -in >> >> /etc/pki/ovirt-engine/private/ca.pem >> >> -enddate -noout >> >> unable to load certificate >> >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start >> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE >> >> >> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017 >> >> >> >> Any ideas? >> >> >> >> Googling surprisingly doesn't come up with much. >> >> >> >> Thank you. >> >> >> >> Regards. >> >> >> >> Neil Wilson. >> >> >> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski >> >> <piotr.kliczewski@gmail.com> wrote: >> >>> >> >>> Neil, >> >>> >> >>> You checked both nodes what about the engine? Can you check engine >> >>> certs? >> >>> You can find more info where they are located here [1]. >> >>> >> >>> Thanks, >> >>> Piotr >> >>> >> >>> [1] >> >>> >> >>> https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine >> >>> >> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: >> >>> > Hi guys, >> >>> > >> >>> > Please could someone assist, my cluster is down and I can't access >> >>> > my >> >>> > vm's >> >>> > to switch some of them back on. >> >>> > >> >>> > I'm seeing the following error in the engine.log however I've >> >>> > checked >> >>> > my >> >>> > certs on my hosts (as some of the goolge results said to check), >> >>> > but >> >>> > the >> >>> > certs haven't expired... >> >>> > >> >>> > >> >>> > 2017-09-21 15:09:45,077 ERROR >> >>> > >> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] >> >>> > (DefaultQuartzScheduler_Worker-4) Command >> >>> > GetCapabilitiesVDSCommand(HostName >> >>> > = node02.mydomain.za, HostId = >> >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, >> >>> > vds=Host[node02.mydomain.za]) execution failed. Exception: >> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received >> >>> > fatal >> >>> > alert: certificate_expired >> >>> > 2017-09-21 15:09:45,086 ERROR >> >>> > >> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVD SCommand] >> >>> > (DefaultQuartzScheduler_Worker-10) Command >> >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = >> >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ node01.mydomain.za]) >> >>> > execution failed. Exception: VDSNetworkException: >> >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert: >> >>> > certificate_expired >> >>> > 2017-09-21 15:09:48,173 ERROR >> >>> > >> >>> > My engine and host info is below... >> >>> > >> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt >> >>> > ovirt-engine-lib-3.4.0-1.el6.noarch >> >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch >> >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch >> >>> > ovirt-engine-3.4.0-1.el6.noarch >> >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch >> >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch >> >>> > ovirt-engine-setup-3.4.0-1.el6.noarch >> >>> > ovirt-host-deploy-1.2.0-1.el6.noarch >> >>> > ovirt-engine-backend-3.4.0-1.el6.noarch >> >>> > ovirt-image-uploader-3.4.0-1.el6.noarch >> >>> > ovirt-engine-tools-3.4.0-1.el6.noarch >> >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch >> >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch >> >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch >> >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch >> >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch >> >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch >> >>> > ovirt-log-collector-3.4.1-1.el6.noarch >> >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch >> >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.no arch >> >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch >> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release >> >>> > CentOS release 6.5 (Final) >> >>> > >> >>> > >> >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >> >>> > -enddate >> >>> > -noout ; date >> >>> > notAfter=May 27 08:36:17 2019 GMT >> >>> > Thu Sep 21 15:18:22 SAST 2017 >> >>> > CentOS release 6.5 (Final) >> >>> > [root@node02 ~]# rpm -qa | grep vdsm >> >>> > vdsm-4.14.6-0.el6.x86_64 >> >>> > vdsm-python-4.14.6-0.el6.x86_64 >> >>> > vdsm-cli-4.14.6-0.el6.noarch >> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >> >>> > >> >>> > >> >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >> >>> > -enddate >> >>> > -noout ; date >> >>> > notAfter=Jun 13 16:09:41 2018 GMT >> >>> > Thu Sep 21 15:18:52 SAST 2017 >> >>> > CentOS release 6.5 (Final) >> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm >> >>> > vdsm-4.14.6-0.el6.x86_64 >> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >> >>> > vdsm-cli-4.14.6-0.el6.noarch >> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >> >>> > vdsm-python-4.14.6-0.el6.x86_64 >> >>> > >> >>> > Please could I have some assistance, I'm rater desperate. >> >>> > >> >>> > Thank you. >> >>> > >> >>> > Regards. >> >>> > >> >>> > Neil Wilson >> >>> > >> >>> > >> >>> > >> >>> > _______________________________________________ >> >>> > Users mailing list >> >>> > Users@ovirt.org >> >>> > http://lists.ovirt.org/mailman/listinfo/users >> >>> > >> >> >> >> >> > > > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

Thank you everyone. I've updated to ovirt-engine-3.5.6.2-1 and this has resolved the problem as it renewed my certs on engine-setup. Much appreciated! Regards. Neil Wilson. On Fri, Sep 22, 2017 at 3:18 PM, Neil <nwilson123@gmail.com> wrote:
Thanks Sandro.
I'll get cracking and report back if it fixed it.
Thanks for all the help everyone.
On Fri, Sep 22, 2017 at 3:14 PM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
2017-09-22 15:07 GMT+02:00 Neil <nwilson123@gmail.com>:
Thanks for the guidance everyone.
I've upgraded my engine now to ovirt-engine-3.4.4-1 but I've still got the same error unfortunately. Below is the output of the upgrade. Should this have fixed the issue or do I need to upgrade to 3.5 etc?
I think you'll need 3.5.4 at least: https://bugzilla.redhat .com/show_bug.cgi?id=1214860
[ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /var/log/ovirt-engine/setup/ov irt-engine-setup-20170922125526-vw5khx.log Version: otopi-1.2.3 (otopi-1.2.3-1.el6) [ INFO ] Stage: Environment packages setup [ INFO ] Yum Downloading: repomdPLa0LXtmp.xml (0%) [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization
--== PRODUCT OPTIONS ==--
--== PACKAGES ==--
[ INFO ] Checking for product updates... Setup has found updates for some packages, do you wish to update them now? (Yes, No) [Yes]: [ INFO ] Checking for an update for Setup...
--== NETWORK CONFIGURATION ==--
[WARNING] Failed to resolve engine01.mydomain.za using DNS, it can be resolved only locally Setup can automatically configure the firewall on this system. Note: automatic configuration of the firewall may overwrite current settings. Do you want Setup to configure the firewall? (Yes, No) [Yes]: no
--== DATABASE CONFIGURATION ==--
--== OVIRT ENGINE CONFIGURATION ==--
Skipping storing options as database already prepared
--== PKI CONFIGURATION ==--
PKI is already configured
--== APACHE CONFIGURATION ==--
--== SYSTEM CONFIGURATION ==--
--== MISC CONFIGURATION ==--
--== END OF CONFIGURATION ==--
[ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [WARNING] Less than 16384MB of memory is available [ INFO ] Cleaning stale zombie tasks
--== CONFIGURATION PREVIEW ==--
Engine database name : engine Engine database secured connection : False Engine database host : localhost Engine database user name : engine Engine database host name validation : False Engine database port : 5432 Datacenter storage type : False Update Firewall : False Configure WebSocket Proxy : True Host FQDN : engine01.mydomain.za Upgrade packages : True
Please confirm installation settings (OK, Cancel) [OK]: [ INFO ] Cleaning async tasks and compensations [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Yum Status: Downloading Packages [ INFO ] Yum Download/Verify: ovirt-engine-3.4.4-1.el6.noarch [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 2.0 M(19%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 4.3 M(41%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 6.3 M(60%) [ INFO ] Yum Downloading: (2/13): ovirt-engine-backend-3.4.4-1.el6.noarch.rpm 8.9 M(85%) [ INFO ] Yum Download/Verify: ovirt-engine-backend-3.4.4-1.el6.noarch [ INFO ] Yum Download/Verify: ovirt-engine-dbscripts-3.4.4-1.el6.noarch (I've taken out all the downloading progress)
[ INFO ] Yum Verify: 26/26: ovirt-engine-backend.noarch 0:3.4.0-1.el6 - ud [ INFO ] Stage: Misc configuration [ INFO ] Backing up database localhost:engine to '/var/lib/ovirt-engine/backups/engine-20170922143709.m_8fr_.dump'. [ INFO ] Updating Engine database schema [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up
--== SUMMARY ==--
[WARNING] Less than 16384MB of memory is available SSH fingerprint: 86:C7:AA:35:45:E9:83:3E:16:C9: 2A:F5:68:52:68:84 Internal CA EE:91:B3:E7:40:D7:DD:A7:DD:77: 9C:3B:D5:A1:E7:BE:E2:C9:8B:AA Web access is enabled at: http://engine01.mydomain.za:80/ovirt-engine https://engine01.mydomain.za:443/ovirt-engine In order to configure firewalld, copy the files from /etc/ovirt-engine/firewalld to /etc/firewalld/services and execute the following commands: firewall-cmd -service ovirt-postgres firewall-cmd -service ovirt-https firewall-cmd -service ovirt-websocket-proxy firewall-cmd -service ovirt-http The following network ports should be opened: tcp:443 tcp:5432 tcp:6100 tcp:80 An example of the required configuration for iptables can be found at: /etc/ovirt-engine/iptables.example
--== END OF SUMMARY ==--
[ INFO ] Starting engine service [ INFO ] Restarting httpd [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ov irt-engine-setup-20170922125526-vw5khx.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/ 20170922143806-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Execution of setup completed successfully
I'm still seeing the following below, in my engine.log and when I log in, all my VM's show as unknown.
2017-09-22 15:06:06,060 ERROR [org.ovirt.engine.core.vdsbrok er.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-57) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za,d2 debdfe-76e7-40cf-a7fd-78a0f50f14d4]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired
Any ideas?
Thanks!
On Fri, Sep 22, 2017 at 11:10 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 22, 2017 at 10:58 AM, Neil <nwilson123@gmail.com> wrote:
Thanks Martin and Piotr,
Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.
I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014
I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?
Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either:
1. Execute engine-setup --offline to skip updates check or 2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site resources.ovirt.org
Thanks for the assistance.
Regards.
Neil Wilson
On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski < piotr.kliczewski@gmail.com> wrote:
On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote: > > > On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote: >> >> Hi Piotr, >> >> Thank you for the information. >> >> It looks like something has expired looking in the server.log now that >> debug is enabled. >> >> 2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version: >> V3 >> 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject: >> CN=engine01.mydomain.za, O=mydomain, C=US >> 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4) >> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 >> 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4) >> 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key: >> Sun RSA public key, 1024 bits >> 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus: >> 966706131850237857720016566132274169225143716493132034132811 213711757321195965137528821713060454503460188878350322233731 259812207539722762942035931744044702655933680916835641105243 164032601213316092139626126181817086803318505413903188689260 54438078223371655800890725486783860059873397983318033852172060923531 >> 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) public >> exponent: 65537 >> 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4) >> Validity: [From: Sun Oct 14 22:26:46 SAST 2012, >> 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4) >> To: Tue Sep 19 18:26:49 SAST 2017] >> 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer: >> CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US >> >> Any idea how I can generate a new one and what cert it is that's expired? > > > It seems that your engine certificate has expired, but AFAIK this > certificate should be automatically renewed during engine-setup. So when did > you execute engine-setup for last time? Any info/warning about this shown > during invocation?
Correct, Martin was a bit faster then me :)
> > Also looking at server.log I found JBoss 7.1.1, so you are using really > ancient oVirt, version, right? > >> >> Please see the attached log for more info. >> >> Thank you so much for your assistance. >> >> Regards. >> >> Neil Wilson. >> >> >> >> >> >> >> On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski >> <piotr.kliczewski@gmail.com> wrote: >>> >>> Neil, >>> >>> It seems that your engine certificate(s) is/are not ok. I would >>> suggest to enable ssl debug in the engine by: >>> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1]. >>> - restart your engine >>> - check your server.log and check what is the issue. >>> >>> Hopefully we will be able to understand what happened in your setup. >>> >>> Thanks, >>> Piotr >>> >>> [1] >>> https://github.com/oVirt/ovirt-engine/blob/master/packaging/ services/ovirt-engine/ovirt-engine.py#L341 >>> >>> On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote: >>> > Further to the logs sent, on the nodes I'm also seeing the following >>> > error >>> > under /var/log/messages... >>> > >>> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with >>> > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C >>> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler >>> > exception#012Traceback >>> > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC.py", >>> > line >>> > 80, in threaded_start#012 self.server.handle_request()#012 File >>> > "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012 >>> > self._handle_request_noblock()#012 File >>> > "/usr/lib64/python2.6/SocketServer.py", line 288, in >>> > _handle_request_noblock#012 request, client_address = >>> > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe rver.py", >>> > line >>> > 456, in get_request#012 return self.socket.accept()#012 File >>> > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line >>> > 136, >>> > in accept#012 raise SSL.SSLError("%s, client %s" % (e, >>> > address[0]))#012SSLError: no certificate returned, client 10.251.193.5 >>> > >>> > Not sure if this is any further help in diagnosing the issue? >>> > >>> > Thanks, any assistance is appreciated. >>> > >>> > Regards. >>> > >>> > Neil Wilson. >>> > >>> > >>> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote: >>> >> >>> >> Hi Piotr, >>> >> >>> >> Thank you for the reply. After sending the email I did go and check >>> >> the >>> >> engine one too.... >>> >> >>> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem >>> >> -enddate >>> >> -noout >>> >> notAfter=Oct 13 16:26:46 2022 GMT >>> >> >>> >> I'm not sure if this one below is meant to verify or if this output is >>> >> expected? >>> >> >>> >> [root@engine01 /]# openssl x509 -in >>> >> /etc/pki/ovirt-engine/private/ca.pem >>> >> -enddate -noout >>> >> unable to load certificate >>> >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start >>> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE >>> >> >>> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017 >>> >> >>> >> Any ideas? >>> >> >>> >> Googling surprisingly doesn't come up with much. >>> >> >>> >> Thank you. >>> >> >>> >> Regards. >>> >> >>> >> Neil Wilson. >>> >> >>> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski >>> >> <piotr.kliczewski@gmail.com> wrote: >>> >>> >>> >>> Neil, >>> >>> >>> >>> You checked both nodes what about the engine? Can you check engine >>> >>> certs? >>> >>> You can find more info where they are located here [1]. >>> >>> >>> >>> Thanks, >>> >>> Piotr >>> >>> >>> >>> [1] >>> >>> >>> >>> https://www.ovirt.org/develop/release-management/features/in fra/pki/#ovirt-engine >>> >>> >>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote: >>> >>> > Hi guys, >>> >>> > >>> >>> > Please could someone assist, my cluster is down and I can't access >>> >>> > my >>> >>> > vm's >>> >>> > to switch some of them back on. >>> >>> > >>> >>> > I'm seeing the following error in the engine.log however I've >>> >>> > checked >>> >>> > my >>> >>> > certs on my hosts (as some of the goolge results said to check), >>> >>> > but >>> >>> > the >>> >>> > certs haven't expired... >>> >>> > >>> >>> > >>> >>> > 2017-09-21 15:09:45,077 ERROR >>> >>> > >>> >>> > [org.ovirt.engine.core.vdsbrok er.vdsbroker.GetCapabilitiesVDSCommand] >>> >>> > (DefaultQuartzScheduler_Worker-4) Command >>> >>> > GetCapabilitiesVDSCommand(HostName >>> >>> > = node02.mydomain.za, HostId = >>> >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, >>> >>> > vds=Host[node02.mydomain.za]) execution failed. Exception: >>> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received >>> >>> > fatal >>> >>> > alert: certificate_expired >>> >>> > 2017-09-21 15:09:45,086 ERROR >>> >>> > >>> >>> > [org.ovirt.engine.core.vdsbrok er.vdsbroker.GetCapabilitiesVDSCommand] >>> >>> > (DefaultQuartzScheduler_Worker-10) Command >>> >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = >>> >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[ node01.mydomain.za]) >>> >>> > execution failed. Exception: VDSNetworkException: >>> >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert: >>> >>> > certificate_expired >>> >>> > 2017-09-21 15:09:48,173 ERROR >>> >>> > >>> >>> > My engine and host info is below... >>> >>> > >>> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt >>> >>> > ovirt-engine-lib-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-setup-plugin-webs ocket-proxy-3.4.0-1.el6.noarch >>> >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch >>> >>> > ovirt-engine-setup-3.4.0-1.el6.noarch >>> >>> > ovirt-host-deploy-1.2.0-1.el6.noarch >>> >>> > ovirt-engine-backend-3.4.0-1.el6.noarch >>> >>> > ovirt-image-uploader-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-tools-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch >>> >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch >>> >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch >>> >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch >>> >>> > ovirt-log-collector-3.4.1-1.el6.noarch >>> >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-setup-plugin-ovir t-engine-common-3.4.0-1.el6.noarch >>> >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch >>> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release >>> >>> > CentOS release 6.5 (Final) >>> >>> > >>> >>> > >>> >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >>> >>> > -enddate >>> >>> > -noout ; date >>> >>> > notAfter=May 27 08:36:17 2019 GMT >>> >>> > Thu Sep 21 15:18:22 SAST 2017 >>> >>> > CentOS release 6.5 (Final) >>> >>> > [root@node02 ~]# rpm -qa | grep vdsm >>> >>> > vdsm-4.14.6-0.el6.x86_64 >>> >>> > vdsm-python-4.14.6-0.el6.x86_64 >>> >>> > vdsm-cli-4.14.6-0.el6.noarch >>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >>> >>> > >>> >>> > >>> >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem >>> >>> > -enddate >>> >>> > -noout ; date >>> >>> > notAfter=Jun 13 16:09:41 2018 GMT >>> >>> > Thu Sep 21 15:18:52 SAST 2017 >>> >>> > CentOS release 6.5 (Final) >>> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm >>> >>> > vdsm-4.14.6-0.el6.x86_64 >>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch >>> >>> > vdsm-cli-4.14.6-0.el6.noarch >>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch >>> >>> > vdsm-python-4.14.6-0.el6.x86_64 >>> >>> > >>> >>> > Please could I have some assistance, I'm rater desperate. >>> >>> > >>> >>> > Thank you. >>> >>> > >>> >>> > Regards. >>> >>> > >>> >>> > Neil Wilson >>> >>> > >>> >>> > >>> >>> > >>> >>> > _______________________________________________ >>> >>> > Users mailing list >>> >>> > Users@ovirt.org >>> >>> > http://lists.ovirt.org/mailman/listinfo/users >>> >>> > >>> >> >>> >> >>> > >> >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >
--
SANDRO BONAZZOLA
ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

2017-09-21 15:26 GMT+02:00 Neil <nwilson123@gmail.com>:
Hi guys,
Please could someone assist, my cluster is down and I can't access my vm's to switch some of them back on.
I'm seeing the following error in the engine.log however I've checked my certs on my hosts (as some of the goolge results said to check), but the certs haven't expired...
2017-09-21 15:09:45,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command GetCapabilitiesVDSCommand(HostName = node02.mydomain.za, HostId = d2debdfe-76e7-40cf-a7fd-78a0f50f14d4, vds=Host[node02.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:45,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-10) Command GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId = b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za]) execution failed. Exception: VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired 2017-09-21 15:09:48,173 ERROR
My engine and host info is below...
[root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt ovirt-engine-lib-3.4.0-1.el6.noarch ovirt-engine-restapi-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch ovirt-engine-3.4.0-1.el6.noarch
People already answered about the certificate expiration. Please note ovirt-engine-3.4.0 is the first release in the 3.4 series which received 4 updates in its lifecycle (latest is 3.4.4, https://www.ovirt.org/develop/release-management/releases/3.4.4/ ) Please consider updating to a supported version as soon as possible.
ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch ovirt-host-deploy-java-1.2.0-1.el6.noarch ovirt-engine-setup-3.4.0-1.el6.noarch ovirt-host-deploy-1.2.0-1.el6.noarch ovirt-engine-backend-3.4.0-1.el6.noarch ovirt-image-uploader-3.4.0-1.el6.noarch ovirt-engine-tools-3.4.0-1.el6.noarch ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch ovirt-engine-cli-3.4.0.5-1.el6.noarch ovirt-engine-setup-base-3.4.0-1.el6.noarch ovirt-iso-uploader-3.4.0-1.el6.noarch ovirt-engine-userportal-3.4.0-1.el6.noarch ovirt-log-collector-3.4.1-1.el6.noarch ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch ovirt-engine-dbscripts-3.4.0-1.el6.noarch [root@engine01 ovirt-engine]# cat /etc/redhat-release CentOS release 6.5 (Final)
[root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=May 27 08:36:17 2019 GMT Thu Sep 21 15:18:22 SAST 2017 CentOS release 6.5 (Final) [root@node02 ~]# rpm -qa | grep vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-python-4.14.6-0.el6.x86_64 vdsm-cli-4.14.6-0.el6.noarch vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch
[root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -enddate -noout ; date notAfter=Jun 13 16:09:41 2018 GMT Thu Sep 21 15:18:52 SAST 2017 CentOS release 6.5 (Final) [root@node01 ~]# rpm -qa | grep -i vdsm vdsm-4.14.6-0.el6.x86_64 vdsm-xmlrpc-4.14.6-0.el6.noarch vdsm-cli-4.14.6-0.el6.noarch vdsm-python-zombiereaper-4.14.6-0.el6.noarch vdsm-python-4.14.6-0.el6.x86_64
Please could I have some assistance, I'm rater desperate.
Thank you.
Regards.
Neil Wilson
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>
participants (4)
-
Martin Perina
-
Neil
-
Piotr Kliczewski
-
Sandro Bonazzola