On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <mperina@redhat.com> wrote:
>
>
> On Fri, Sep 22, 2017 at 10:18 AM, Neil <nwilson123@gmail.com> wrote:
>>
>> Hi Piotr,
>>
>> Thank you for the information.
>>
>> It looks like something has expired looking in the server.log now that
>> debug is enabled.
>>
>> 2017-09-22 09:35:26,462 INFO  [stdout] (MSC service thread 1-4)   Version:
>> V3
>> 2017-09-22 09:35:26,464 INFO  [stdout] (MSC service thread 1-4)   Subject:
>> CN=engine01.mydomain.za, O=mydomain, C=US
>> 2017-09-22 09:35:26,467 INFO  [stdout] (MSC service thread 1-4)
>> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5
>> 2017-09-22 09:35:26,471 INFO  [stdout] (MSC service thread 1-4)
>> 2017-09-22 09:35:26,472 INFO  [stdout] (MSC service thread 1-4)   Key:
>> Sun RSA public key, 1024 bits
>> 2017-09-22 09:35:26,474 INFO  [stdout] (MSC service thread 1-4)   modulus:
>> 96670613185023785772001656613227416922514371649313203413281121371175732119596513752882171306045450346018887835032223373125981220753972276294203593174404470265593368091683564110524316403260121331609213962612618181708680331850541390318868926054438078223371655800890725486783860059873397983318033852172060923531
>> 2017-09-22 09:35:26,476 INFO  [stdout] (MSC service thread 1-4)   public
>> exponent: 65537
>> 2017-09-22 09:35:26,477 INFO  [stdout] (MSC service thread 1-4)
>> Validity: [From: Sun Oct 14 22:26:46 SAST 2012,
>> 2017-09-22 09:35:26,478 INFO  [stdout] (MSC service thread 1-4)
>> To: Tue Sep 19 18:26:49 SAST 2017]
>> 2017-09-22 09:35:26,479 INFO  [stdout] (MSC service thread 1-4)   Issuer:
>> CN=CA-engine01.mydomain.za.47472, O=mydomain, C=US
>>
>> Any idea how I can generate a new one and what cert it is that's expired?
>
>
> It seems that your engine certificate has expired, but AFAIK this
> certificate should be automatically renewed during engine-setup. So when did
> you execute engine-setup for last time? Any info/warning about this shown
> during invocation?

Correct, Martin was a bit faster then me :)

>
> Also looking at server.log I found JBoss 7.1.1, so you are using really
> ancient oVirt, version, right?
>
>>
>> Please see the attached log for more info.
>>
>> Thank you so much for your assistance.
>>
>> Regards.
>>
>> Neil Wilson.
>>
>>
>>
>>
>>
>>
>> On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski
>> <piotr.kliczewski@gmail.com> wrote:
>>>
>>> Neil,
>>>
>>> It seems that your engine certificate(s) is/are not ok. I would
>>> suggest to enable ssl debug in the engine by:
>>> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1].
>>> - restart your engine
>>> - check your server.log and check what is the issue.
>>>
>>> Hopefully we will be able to understand what happened in your setup.
>>>
>>> Thanks,
>>> Piotr
>>>
>>> [1]
>>> https://github.com/oVirt/ovirt-engine/blob/master/packaging/services/ovirt-engine/ovirt-engine.py#L341
>>>
>>> On Thu, Sep 21, 2017 at 4:42 PM, Neil <nwilson123@gmail.com> wrote:
>>> > Further to the logs sent, on the nodes I'm also seeing the following
>>> > error
>>> > under /var/log/messages...
>>> >
>>> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with
>>> > subject "/C=US/O=UKDM/CN=engine01.mydomain.za"^C
>>> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler
>>> > exception#012Traceback
>>> > (most recent call last):#012  File "/usr/share/vdsm/BindingXMLRPC.py",
>>> > line
>>> > 80, in threaded_start#012    self.server.handle_request()#012  File
>>> > "/usr/lib64/python2.6/SocketServer.py", line 278, in handle_request#012
>>> > self._handle_request_noblock()#012  File
>>> > "/usr/lib64/python2.6/SocketServer.py", line 288, in
>>> > _handle_request_noblock#012    request, client_address =
>>> > self.get_request()#012  File "/usr/lib64/python2.6/SocketServer.py",
>>> > line
>>> > 456, in get_request#012    return self.socket.accept()#012  File
>>> > "/usr/lib64/python2.6/site-packages/vdsm/SecureXMLRPCServer.py", line
>>> > 136,
>>> > in accept#012    raise SSL.SSLError("%s, client %s" % (e,
>>> > address[0]))#012SSLError: no certificate returned, client 10.251.193.5
>>> >
>>> > Not sure if this is any further help in diagnosing the issue?
>>> >
>>> > Thanks, any assistance is appreciated.
>>> >
>>> > Regards.
>>> >
>>> > Neil Wilson.
>>> >
>>> >
>>> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <nwilson123@gmail.com> wrote:
>>> >>
>>> >> Hi Piotr,
>>> >>
>>> >> Thank you for the reply. After sending the email I did go and check
>>> >> the
>>> >> engine one too....
>>> >>
>>> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem
>>> >> -enddate
>>> >> -noout
>>> >> notAfter=Oct 13 16:26:46 2022 GMT
>>> >>
>>> >> I'm not sure if this one below is meant to verify or if this output is
>>> >> expected?
>>> >>
>>> >> [root@engine01 /]# openssl x509 -in
>>> >> /etc/pki/ovirt-engine/private/ca.pem
>>> >> -enddate -noout
>>> >> unable to load certificate
>>> >> 140642165552968:error:0906D06C:PEM routines:PEM_read_bio:no start
>>> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
>>> >>
>>> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017
>>> >>
>>> >> Any ideas?
>>> >>
>>> >> Googling surprisingly doesn't come up with much.
>>> >>
>>> >> Thank you.
>>> >>
>>> >> Regards.
>>> >>
>>> >> Neil Wilson.
>>> >>
>>> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski
>>> >> <piotr.kliczewski@gmail.com> wrote:
>>> >>>
>>> >>> Neil,
>>> >>>
>>> >>> You checked both nodes what about the engine? Can you check engine
>>> >>> certs?
>>> >>> You can find more info where they are located here [1].
>>> >>>
>>> >>> Thanks,
>>> >>> Piotr
>>> >>>
>>> >>> [1]
>>> >>>
>>> >>> https://www.ovirt.org/develop/release-management/features/infra/pki/#ovirt-engine
>>> >>>
>>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <nwilson123@gmail.com> wrote:
>>> >>> > Hi guys,
>>> >>> >
>>> >>> > Please could someone assist, my cluster is down and I can't access
>>> >>> > my
>>> >>> > vm's
>>> >>> > to switch some of them back on.
>>> >>> >
>>> >>> > I'm seeing the following error in the engine.log however I've
>>> >>> > checked
>>> >>> > my
>>> >>> > certs on my hosts (as some of the goolge results said to check),
>>> >>> > but
>>> >>> > the
>>> >>> > certs haven't expired...
>>> >>> >
>>> >>> >
>>> >>> > 2017-09-21 15:09:45,077 ERROR
>>> >>> >
>>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>> >>> > (DefaultQuartzScheduler_Worker-4) Command
>>> >>> > GetCapabilitiesVDSCommand(HostName
>>> >>> > = node02.mydomain.za, HostId =
>>> >>> > d2debdfe-76e7-40cf-a7fd-78a0f50f14d4,
>>> >>> > vds=Host[node02.mydomain.za]) execution failed. Exception:
>>> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeException: Received
>>> >>> > fatal
>>> >>> > alert: certificate_expired
>>> >>> > 2017-09-21 15:09:45,086 ERROR
>>> >>> >
>>> >>> > [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
>>> >>> > (DefaultQuartzScheduler_Worker-10) Command
>>> >>> > GetCapabilitiesVDSCommand(HostName = node01.mydomain.za, HostId =
>>> >>> > b108549c-1700-11e2-b936-9f5243b8ce13, vds=Host[node01.mydomain.za])
>>> >>> > execution failed. Exception: VDSNetworkException:
>>> >>> > javax.net.ssl.SSLHandshakeException: Received fatal alert:
>>> >>> > certificate_expired
>>> >>> > 2017-09-21 15:09:48,173 ERROR
>>> >>> >
>>> >>> > My engine and host info is below...
>>> >>> >
>>> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt
>>> >>> > ovirt-engine-lib-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-restapi-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-plugin-ovirt-engine-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-plugin-websocket-proxy-3.4.0-1.el6.noarch
>>> >>> > ovirt-host-deploy-java-1.2.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-3.4.0-1.el6.noarch
>>> >>> > ovirt-host-deploy-1.2.0-1.el6.noarch
>>> >>> > ovirt-engine-backend-3.4.0-1.el6.noarch
>>> >>> > ovirt-image-uploader-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-tools-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-sdk-python-3.4.0.7-1.el6.noarch
>>> >>> > ovirt-engine-webadmin-portal-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-cli-3.4.0.5-1.el6.noarch
>>> >>> > ovirt-engine-setup-base-3.4.0-1.el6.noarch
>>> >>> > ovirt-iso-uploader-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-userportal-3.4.0-1.el6.noarch
>>> >>> > ovirt-log-collector-3.4.1-1.el6.noarch
>>> >>> > ovirt-engine-websocket-proxy-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-setup-plugin-ovirt-engine-common-3.4.0-1.el6.noarch
>>> >>> > ovirt-engine-dbscripts-3.4.0-1.el6.noarch
>>> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release
>>> >>> > CentOS release 6.5 (Final)
>>> >>> >
>>> >>> >
>>> >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem
>>> >>> > -enddate
>>> >>> > -noout ; date
>>> >>> > notAfter=May 27 08:36:17 2019 GMT
>>> >>> > Thu Sep 21 15:18:22 SAST 2017
>>> >>> > CentOS release 6.5 (Final)
>>> >>> > [root@node02 ~]# rpm -qa | grep vdsm
>>> >>> > vdsm-4.14.6-0.el6.x86_64
>>> >>> > vdsm-python-4.14.6-0.el6.x86_64
>>> >>> > vdsm-cli-4.14.6-0.el6.noarch
>>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch
>>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch
>>> >>> >
>>> >>> >
>>> >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem
>>> >>> > -enddate
>>> >>> > -noout ; date
>>> >>> > notAfter=Jun 13 16:09:41 2018 GMT
>>> >>> > Thu Sep 21 15:18:52 SAST 2017
>>> >>> > CentOS release 6.5 (Final)
>>> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm
>>> >>> > vdsm-4.14.6-0.el6.x86_64
>>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarch
>>> >>> > vdsm-cli-4.14.6-0.el6.noarch
>>> >>> > vdsm-python-zombiereaper-4.14.6-0.el6.noarch
>>> >>> > vdsm-python-4.14.6-0.el6.x86_64
>>> >>> >
>>> >>> > Please could I have some assistance, I'm rater desperate.
>>> >>> >
>>> >>> > Thank you.
>>> >>> >
>>> >>> > Regards.
>>> >>> >
>>> >>> > Neil Wilson
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > Users mailing list
>>> >>> > Users@ovirt.org
>>> >>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >>> >
>>> >>
>>> >>
>>> >
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>