<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 22, 2017 at 10:58 AM, Neil <span dir="ltr"><<a href="mailto:nwilson123@gmail.com" target="_blank">nwilson123@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Martin and Piotr,<div><br></div><div>Correct, this was a very old installation from the old drey repo that was upgraded gradually over the years.</div><div><br></div><div>I have tried engine-setup yesterday, prior to this looking under /var/log/ovirt-engine/setup it looks like 2014</div><div><br></div><div>I've attached a log of the output of running it now, looks like a repo issue with trying to upgrade to the latest 3.4.x release, but not sure what else to look for?</div></div></blockquote><div><br><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">Hmm, it's so ancient version that oVirt 3.4 mirrors are probably not working anymore. You can either:<br><br></div><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">1. Execute engine-setup --offline to skip updates check or<br></div><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">2. Edit /etc/yum.repos.d/ovirt*.conf files and switch from mirrors to main site <a href="http://resources.ovirt.org">resources.ovirt.org</a><br><br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Thanks for the assistance.</div><div><br></div><div>Regards.</div><div><br></div><div>Neil Wilson</div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 22, 2017 at 10:38 AM, Piotr Kliczewski <span dir="ltr"><<a href="mailto:piotr.kliczewski@gmail.com" target="_blank">piotr.kliczewski@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_-7948330153316553372gmail-HOEnZb"><div class="m_-7948330153316553372gmail-h5">On Fri, Sep 22, 2017 at 10:35 AM, Martin Perina <<a href="mailto:mperina@redhat.com" target="_blank">mperina@redhat.com</a>> wrote:<br>
><br>
><br>
> On Fri, Sep 22, 2017 at 10:18 AM, Neil <<a href="mailto:nwilson123@gmail.com" target="_blank">nwilson123@gmail.com</a>> wrote:<br>
>><br>
>> Hi Piotr,<br>
>><br>
>> Thank you for the information.<br>
>><br>
>> It looks like something has expired looking in the server.log now that<br>
>> debug is enabled.<br>
>><br>
>> 2017-09-22 09:35:26,462 INFO [stdout] (MSC service thread 1-4) Version:<br>
>> V3<br>
>> 2017-09-22 09:35:26,464 INFO [stdout] (MSC service thread 1-4) Subject:<br>
>> CN=<a href="http://engine01.mydomain.za" rel="noreferrer" target="_blank">engine01.mydomain.za</a>, O=mydomain, C=US<br>
>> 2017-09-22 09:35:26,467 INFO [stdout] (MSC service thread 1-4)<br>
>> Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5<br>
>> 2017-09-22 09:35:26,471 INFO [stdout] (MSC service thread 1-4)<br>
>> 2017-09-22 09:35:26,472 INFO [stdout] (MSC service thread 1-4) Key:<br>
>> Sun RSA public key, 1024 bits<br>
>> 2017-09-22 09:35:26,474 INFO [stdout] (MSC service thread 1-4) modulus:<br>
>> 966706131850237857720016566132<wbr>274169225143716493132034132811<wbr>213711757321195965137528821713<wbr>060454503460188878350322233731<wbr>259812207539722762942035931744<wbr>044702655933680916835641105243<wbr>164032601213316092139626126181<wbr>817086803318505413903188689260<wbr>544380782233716558008907254867<wbr>838600598733979833180338521720<wbr>60923531<br>
>> 2017-09-22 09:35:26,476 INFO [stdout] (MSC service thread 1-4) public<br>
>> exponent: 65537<br>
>> 2017-09-22 09:35:26,477 INFO [stdout] (MSC service thread 1-4)<br>
>> Validity: [From: Sun Oct 14 22:26:46 SAST 2012,<br>
>> 2017-09-22 09:35:26,478 INFO [stdout] (MSC service thread 1-4)<br>
>> To: Tue Sep 19 18:26:49 SAST 2017]<br>
>> 2017-09-22 09:35:26,479 INFO [stdout] (MSC service thread 1-4) Issuer:<br>
>> CN=<a href="http://CA-engine01.mydomain.za" target="_blank">CA-engine01.mydomain.za</a>.474<wbr>72, O=mydomain, C=US<br>
>><br>
>> Any idea how I can generate a new one and what cert it is that's expired?<br>
><br>
><br>
> It seems that your engine certificate has expired, but AFAIK this<br>
> certificate should be automatically renewed during engine-setup. So when did<br>
> you execute engine-setup for last time? Any info/warning about this shown<br>
> during invocation?<br>
<br>
</div></div>Correct, Martin was a bit faster then me :)<br>
<div class="m_-7948330153316553372gmail-HOEnZb"><div class="m_-7948330153316553372gmail-h5"><br>
><br>
> Also looking at server.log I found JBoss 7.1.1, so you are using really<br>
> ancient oVirt, version, right?<br>
><br>
>><br>
>> Please see the attached log for more info.<br>
>><br>
>> Thank you so much for your assistance.<br>
>><br>
>> Regards.<br>
>><br>
>> Neil Wilson.<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Thu, Sep 21, 2017 at 8:41 PM, Piotr Kliczewski<br>
>> <<a href="mailto:piotr.kliczewski@gmail.com" target="_blank">piotr.kliczewski@gmail.com</a>> wrote:<br>
>>><br>
>>> Neil,<br>
>>><br>
>>> It seems that your engine certificate(s) is/are not ok. I would<br>
>>> suggest to enable ssl debug in the engine by:<br>
>>> - add '-Djavax.net.debug=all' to ovirt-engine.py file here [1].<br>
>>> - restart your engine<br>
>>> - check your server.log and check what is the issue.<br>
>>><br>
>>> Hopefully we will be able to understand what happened in your setup.<br>
>>><br>
>>> Thanks,<br>
>>> Piotr<br>
>>><br>
>>> [1]<br>
>>> <a href="https://github.com/oVirt/ovirt-engine/blob/master/packaging/services/ovirt-engine/ovirt-engine.py#L341" rel="noreferrer" target="_blank">https://github.com/oVirt/ovirt<wbr>-engine/blob/master/packaging/<wbr>services/ovirt-engine/ovirt-<wbr>engine.py#L341</a><br>
>>><br>
>>> On Thu, Sep 21, 2017 at 4:42 PM, Neil <<a href="mailto:nwilson123@gmail.com" target="_blank">nwilson123@gmail.com</a>> wrote:<br>
>>> > Further to the logs sent, on the nodes I'm also seeing the following<br>
>>> > error<br>
>>> > under /var/log/messages...<br>
>>> ><br>
>>> > Sep 20 03:43:12 node01 vdsm root ERROR invalid client certificate with<br>
>>> > subject "/C=US/O=UKDM/CN=<a href="http://engine01.mydomain.za" rel="noreferrer" target="_blank">engine01.mydo<wbr>main.za</a>"^C<br>
>>> > Sep 20 03:43:12 node01 vdsm vds ERROR xml-rpc handler<br>
>>> > exception#012Traceback<br>
>>> > (most recent call last):#012 File "/usr/share/vdsm/BindingXMLRPC<wbr>.py",<br>
>>> > line<br>
>>> > 80, in threaded_start#012 self.server.handle_request()#0<wbr>12 File<br>
>>> > "/usr/lib64/python2.6/SocketSe<wbr>rver.py", line 278, in handle_request#012<br>
>>> > self._handle_request_noblock()<wbr>#012 File<br>
>>> > "/usr/lib64/python2.6/SocketSe<wbr>rver.py", line 288, in<br>
>>> > _handle_request_noblock#012 request, client_address =<br>
>>> > self.get_request()#012 File "/usr/lib64/python2.6/SocketSe<wbr>rver.py",<br>
>>> > line<br>
>>> > 456, in get_request#012 return self.socket.accept()#012 File<br>
>>> > "/usr/lib64/python2.6/site-pac<wbr>kages/vdsm/SecureXMLRPCServer.<wbr>py", line<br>
>>> > 136,<br>
>>> > in accept#012 raise SSL.SSLError("%s, client %s" % (e,<br>
>>> > address[0]))#012SSLError: no certificate returned, client 10.251.193.5<br>
>>> ><br>
>>> > Not sure if this is any further help in diagnosing the issue?<br>
>>> ><br>
>>> > Thanks, any assistance is appreciated.<br>
>>> ><br>
>>> > Regards.<br>
>>> ><br>
>>> > Neil Wilson.<br>
>>> ><br>
>>> ><br>
>>> > On Thu, Sep 21, 2017 at 4:31 PM, Neil <<a href="mailto:nwilson123@gmail.com" target="_blank">nwilson123@gmail.com</a>> wrote:<br>
>>> >><br>
>>> >> Hi Piotr,<br>
>>> >><br>
>>> >> Thank you for the reply. After sending the email I did go and check<br>
>>> >> the<br>
>>> >> engine one too....<br>
>>> >><br>
>>> >> [root@engine01 /]# openssl x509 -in /etc/pki/ovirt-engine/ca.pem<br>
>>> >> -enddate<br>
>>> >> -noout<br>
>>> >> notAfter=Oct 13 16:26:46 2022 GMT<br>
>>> >><br>
>>> >> I'm not sure if this one below is meant to verify or if this output is<br>
>>> >> expected?<br>
>>> >><br>
>>> >> [root@engine01 /]# openssl x509 -in<br>
>>> >> /etc/pki/ovirt-engine/private/<wbr>ca.pem<br>
>>> >> -enddate -noout<br>
>>> >> unable to load certificate<br>
>>> >> 140642165552968:error:0906D06C<wbr>:PEM routines:PEM_read_bio:no start<br>
>>> >> line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE<br>
>>> >><br>
>>> >> My date is correct too Thu Sep 21 16:30:15 SAST 2017<br>
>>> >><br>
>>> >> Any ideas?<br>
>>> >><br>
>>> >> Googling surprisingly doesn't come up with much.<br>
>>> >><br>
>>> >> Thank you.<br>
>>> >><br>
>>> >> Regards.<br>
>>> >><br>
>>> >> Neil Wilson.<br>
>>> >><br>
>>> >> On Thu, Sep 21, 2017 at 4:16 PM, Piotr Kliczewski<br>
>>> >> <<a href="mailto:piotr.kliczewski@gmail.com" target="_blank">piotr.kliczewski@gmail.com</a>> wrote:<br>
>>> >>><br>
>>> >>> Neil,<br>
>>> >>><br>
>>> >>> You checked both nodes what about the engine? Can you check engine<br>
>>> >>> certs?<br>
>>> >>> You can find more info where they are located here [1].<br>
>>> >>><br>
>>> >>> Thanks,<br>
>>> >>> Piotr<br>
>>> >>><br>
>>> >>> [1]<br>
>>> >>><br>
>>> >>> <a href="https://www.ovirt.org/develop/release-management/features/infra/pki/#ovirt-engine" rel="noreferrer" target="_blank">https://www.ovirt.org/develop/<wbr>release-management/features/in<wbr>fra/pki/#ovirt-engine</a><br>
>>> >>><br>
>>> >>> On Thu, Sep 21, 2017 at 3:26 PM, Neil <<a href="mailto:nwilson123@gmail.com" target="_blank">nwilson123@gmail.com</a>> wrote:<br>
>>> >>> > Hi guys,<br>
>>> >>> ><br>
>>> >>> > Please could someone assist, my cluster is down and I can't access<br>
>>> >>> > my<br>
>>> >>> > vm's<br>
>>> >>> > to switch some of them back on.<br>
>>> >>> ><br>
>>> >>> > I'm seeing the following error in the engine.log however I've<br>
>>> >>> > checked<br>
>>> >>> > my<br>
>>> >>> > certs on my hosts (as some of the goolge results said to check),<br>
>>> >>> > but<br>
>>> >>> > the<br>
>>> >>> > certs haven't expired...<br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> > 2017-09-21 15:09:45,077 ERROR<br>
>>> >>> ><br>
>>> >>> > [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand]<br>
>>> >>> > (DefaultQuartzScheduler_Worker<wbr>-4) Command<br>
>>> >>> > GetCapabilitiesVDSCommand(Host<wbr>Name<br>
>>> >>> > = <a href="http://node02.mydomain.za" rel="noreferrer" target="_blank">node02.mydomain.za</a>, HostId =<br>
>>> >>> > d2debdfe-76e7-40cf-a7fd-78a0f5<wbr>0f14d4,<br>
>>> >>> > vds=Host[<a href="http://node02.mydomain.za" rel="noreferrer" target="_blank">node02.mydomain.za</a>]) execution failed. Exception:<br>
>>> >>> > VDSNetworkException: javax.net.ssl.SSLHandshakeExce<wbr>ption: Received<br>
>>> >>> > fatal<br>
>>> >>> > alert: certificate_expired<br>
>>> >>> > 2017-09-21 15:09:45,086 ERROR<br>
>>> >>> ><br>
>>> >>> > [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.GetCapabilitiesVD<wbr>SCommand]<br>
>>> >>> > (DefaultQuartzScheduler_Worker<wbr>-10) Command<br>
>>> >>> > GetCapabilitiesVDSCommand(Host<wbr>Name = <a href="http://node01.mydomain.za" rel="noreferrer" target="_blank">node01.mydomain.za</a>, HostId =<br>
>>> >>> > b108549c-1700-11e2-b936-9f5243<wbr>b8ce13, vds=Host[<a href="http://node01.mydomain.za" rel="noreferrer" target="_blank">node01.mydomain.za</a>])<br>
>>> >>> > execution failed. Exception: VDSNetworkException:<br>
>>> >>> > javax.net.ssl.SSLHandshakeExce<wbr>ption: Received fatal alert:<br>
>>> >>> > certificate_expired<br>
>>> >>> > 2017-09-21 15:09:48,173 ERROR<br>
>>> >>> ><br>
>>> >>> > My engine and host info is below...<br>
>>> >>> ><br>
>>> >>> > [root@engine01 ovirt-engine]# rpm -qa | grep -i ovirt<br>
>>> >>> > ovirt-engine-lib-3.4.0-1.el6.n<wbr>oarch<br>
>>> >>> > ovirt-engine-restapi-3.4.0-1.e<wbr>l6.noarch<br>
>>> >>> > ovirt-engine-setup-plugin-ovir<wbr>t-engine-3.4.0-1.el6.noarch<br>
>>> >>> > ovirt-engine-3.4.0-1.el6.noarc<wbr>h<br>
>>> >>> > ovirt-engine-setup-plugin-webs<wbr>ocket-proxy-3.4.0-1.el6.noarch<br>
>>> >>> > ovirt-host-deploy-java-1.2.0-1<wbr>.el6.noarch<br>
>>> >>> > ovirt-engine-setup-3.4.0-1.el6<wbr>.noarch<br>
>>> >>> > ovirt-host-deploy-1.2.0-1.el6.<wbr>noarch<br>
>>> >>> > ovirt-engine-backend-3.4.0-1.e<wbr>l6.noarch<br>
>>> >>> > ovirt-image-uploader-3.4.0-1.e<wbr>l6.noarch<br>
>>> >>> > ovirt-engine-tools-3.4.0-1.el6<wbr>.noarch<br>
>>> >>> > ovirt-engine-sdk-python-3.4.0.<wbr>7-1.el6.noarch<br>
>>> >>> > ovirt-engine-webadmin-portal-3<wbr>.4.0-1.el6.noarch<br>
>>> >>> > ovirt-engine-cli-3.4.0.5-1.el6<wbr>.noarch<br>
>>> >>> > ovirt-engine-setup-base-3.4.0-<wbr>1.el6.noarch<br>
>>> >>> > ovirt-iso-uploader-3.4.0-1.el6<wbr>.noarch<br>
>>> >>> > ovirt-engine-userportal-3.4.0-<wbr>1.el6.noarch<br>
>>> >>> > ovirt-log-collector-3.4.1-1.el<wbr>6.noarch<br>
>>> >>> > ovirt-engine-websocket-proxy-3<wbr>.4.0-1.el6.noarch<br>
>>> >>> > ovirt-engine-setup-plugin-ovir<wbr>t-engine-common-3.4.0-1.el6.<wbr>noarch<br>
>>> >>> > ovirt-engine-dbscripts-3.4.0-1<wbr>.el6.noarch<br>
>>> >>> > [root@engine01 ovirt-engine]# cat /etc/redhat-release<br>
>>> >>> > CentOS release 6.5 (Final)<br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> > [root@node02 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.p<wbr>em<br>
>>> >>> > -enddate<br>
>>> >>> > -noout ; date<br>
>>> >>> > notAfter=May 27 08:36:17 2019 GMT<br>
>>> >>> > Thu Sep 21 15:18:22 SAST 2017<br>
>>> >>> > CentOS release 6.5 (Final)<br>
>>> >>> > [root@node02 ~]# rpm -qa | grep vdsm<br>
>>> >>> > vdsm-4.14.6-0.el6.x86_64<br>
>>> >>> > vdsm-python-4.14.6-0.el6.x86_6<wbr>4<br>
>>> >>> > vdsm-cli-4.14.6-0.el6.noarch<br>
>>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarc<wbr>h<br>
>>> >>> > vdsm-python-zombiereaper-4.14.<wbr>6-0.el6.noarch<br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> > [root@node01 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.p<wbr>em<br>
>>> >>> > -enddate<br>
>>> >>> > -noout ; date<br>
>>> >>> > notAfter=Jun 13 16:09:41 2018 GMT<br>
>>> >>> > Thu Sep 21 15:18:52 SAST 2017<br>
>>> >>> > CentOS release 6.5 (Final)<br>
>>> >>> > [root@node01 ~]# rpm -qa | grep -i vdsm<br>
>>> >>> > vdsm-4.14.6-0.el6.x86_64<br>
>>> >>> > vdsm-xmlrpc-4.14.6-0.el6.noarc<wbr>h<br>
>>> >>> > vdsm-cli-4.14.6-0.el6.noarch<br>
>>> >>> > vdsm-python-zombiereaper-4.14.<wbr>6-0.el6.noarch<br>
>>> >>> > vdsm-python-4.14.6-0.el6.x86_6<wbr>4<br>
>>> >>> ><br>
>>> >>> > Please could I have some assistance, I'm rater desperate.<br>
>>> >>> ><br>
>>> >>> > Thank you.<br>
>>> >>> ><br>
>>> >>> > Regards.<br>
>>> >>> ><br>
>>> >>> > Neil Wilson<br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> ><br>
>>> >>> > ______________________________<wbr>_________________<br>
>>> >>> > Users mailing list<br>
>>> >>> > <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
>>> >>> > <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
>>> >>> ><br>
>>> >><br>
>>> >><br>
>>> ><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> Users mailing list<br>
>> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
>><br>
><br>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div>