On Mon, Jun 24, 2019 at 7:56 PM Stefano Danzi <s.danzi@hawai.it> wrote:

I've found that this issue is related to:

https://bugzilla.redhat.com/show_bug.cgi?id=1648190

Are you sure? That bug is about an old cert, generated by an old version, likely before we fixed bug 1210486 (even though it's not mentioned in above bug).

But i've no idea how fix it....

Il 24/06/2019 18:19, Stefano Danzi ha scritto:

...

I've just upgraded my test environment from ovirt 4.2 to 4.3.4.

Was it installed as 4.2, or upgraded? From which first version?

...

System has only one host (Centos 7.6.1810) and run a self hosted engine.

After upgrade I'm not able to run vdsmd (and so hosted engine....)

Above the error in log:

journalctl -xe

-- L'unità libvirtd.service ha iniziato la fase di avvio. giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: info : libvirt version: 4.5.0, package: 10.el7_6.12 (CentOS BuildSystem <http://bugs.centos.org>, 2019-06-20-15:01:15, x86-01.bsys. giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: info : hostname: ovirt01.hawai.lan giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: error : virNetTLSContextLoadCertFromFile:513 : Unable to import server certificate /etc/pki/vdsm/certs/vdsmcert.pem

Did you check this file? Does it exist? ls -l /etc/pki/vdsm/certs/vdsmcert.pem Can vdsm user read it? su - vdsm -s /bin/bash -c 'cat /etc/pki/vdsm/certs/vdsmcert.pem > /dev/null' Please check/share output of: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text Thanks and best regards,

...

giu 24 18:09:17 ovirt01.hawai.lan systemd[1]: libvirtd.service: main process exited, code=exited, status=6/NOTCONFIGURED giu 24 18:09:17 ovirt01.hawai.lan systemd[1]: Failed to start Virtualization daemon. -- Subject: L'unità libvirtd.service è fallita _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MAP4TPH7UAGBFL...

---

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ADFJRSR4BDGD5X...

-- Didi

On Tue, Jun 25, 2019 at 10:26 AM Stefano Danzi <s.danzi@hawai.it> wrote:

Il 25/06/2019 08:27, Yedidyah Bar David ha scritto:

...

On Mon, Jun 24, 2019 at 7:56 PM Stefano Danzi <s.danzi@hawai.it> wrote:

...

I've found that this issue is related to:

https://bugzilla.redhat.com/show_bug.cgi?id=1648190 Are you sure?

That bug is about an old cert, generated by an old version, likely before we fixed bug 1210486 (even though it's not mentioned in above bug).

Yes! Malformed "Not Before" date/time in certs

...

...

But i've no idea how fix it....

Il 24/06/2019 18:19, Stefano Danzi ha scritto:

...

I've just upgraded my test environment from ovirt 4.2 to 4.3.4. Was it installed as 4.2, or upgraded? From which first version?

I don't remember the first installed version. Maybe 4.0... I always upgraded the original installation.

...

...

...

System has only one host (Centos 7.6.1810) and run a self hosted engine.

After upgrade I'm not able to run vdsmd (and so hosted engine....)

Above the error in log:

journalctl -xe

-- L'unità libvirtd.service ha iniziato la fase di avvio. giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: info : libvirt version: 4.5.0, package: 10.el7_6.12 (CentOS BuildSystem <http://bugs.centos.org>, 2019-06-20-15:01:15, x86-01.bsys. giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: info : hostname: ovirt01.hawai.lan giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: error : virNetTLSContextLoadCertFromFile:513 : Unable to import server certificate /etc/pki/vdsm/certs/vdsmcert.pem Did you check this file? Does it exist?

ls -l /etc/pki/vdsm/certs/vdsmcert.pem

Can vdsm user read it?

su - vdsm -s /bin/bash -c 'cat /etc/pki/vdsm/certs/vdsmcert.pem > /dev/null'

Please check/share output of:

openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text

Thanks and best regards,

vdsm can read vdsmcert. The problem is "Not Before" date:

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text' Certificate: Data: Version: 3 (0x2) Serial Number: 4102 (0x1006) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=hawai.lan, CN=ovirtbk-sheng.hawai.lan.63272 Validity Not Before: Feb 4 08:36:07 2015 Not After : Feb 4 08:36:07 2020 GMT [CUT]

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text' Certificate: Data: Version: 3 (0x2) Serial Number: 4096 (0x1000) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=hawai.lan, CN=ovirtbk-sheng.hawai.lan.63272 Validity Not Before: Feb 4 00:06:25 2015 Not After : Feb 2 00:06:25 2025 GMT

OK :-( So it will be rather difficult to fix. You should have been prompted by engine-setup long ago to renew PKI, weren't you? And when you did, didn't you have to reinstall (or Re- Enroll Certificates, in later versions) all hosts? Anyway: If at all possible, please try to downgrade whatever upgrade that caused this to fail. You can check 'yum history', 'yum history info $ID', 'yum history undo $ID'. Then start your engine vm, start the engine, re-install or re-enroll-certs all hosts. See also: https://www.ovirt.org/develop/release-management/releases/3.5.4/#pki Then upgrade again what you downgraded. If that's impossible, it will be harder. I can think of two choices: 1. Consider the engine is completely dead and reinstall everything from scratch. Hopefully, attaching to the existing storage domains and importing all VMs will not be too hard and will not loose too much information. Alternatively, if you have an engine-backup backup, you can try restore from it. hosted-engine in recent versions can do this mostly-automatically. Search the web for "hosted-engine --restore-from-file". 2. Try to manually fix. Something like: - Find the image of the engine vm on the hosted-engine storage - Use some means to "edit" it - e.g. guestfish (but there are also older, less comfortable means - e.g. copy the image elsewhere and start a new kvm VM from it, or something like that). Assuming you manage to get to some environment that lets you run commands inside the engine vm image, in its context: - I do not find a csr for the vdsm key on a host I am checking. Assuming you don't either, you should generate one from its private key. So do this on the host (not engine): openssl req -new -days 365 -key /etc/pki/vdsm/keys/vdsmkey.pem -out /tmp/vdsm.req -batch -subj / Somehow copy /tmp/vdsm.req to the engine machine to e.g. /etc/pki/ovirt-engine/requests/new-host1.req Run on the engine machine something like: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=new-host1 --subject=/O=$ORGANIZATION_NAME/CN=$COMMON_NAME --days=1825 Then copy from the engine machine /etc/pki/ovirt-engine/certs/new-host1.pub to the host at all the places that have copies of the cert. I think these are: /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem But better check first with grep/find (and of course backup beforehand). Then try to start vdsm, and if it works start the engine vm. If all goes well, reinstall or re-enroll-certs on all hosts. Good luck and best regards, -- Didi

On Tue, Jun 25, 2019 at 12:28 PM Stefano Danzi <s.danzi@hawai.it> wrote:

Il 25/06/2019 10:08, Yedidyah Bar David ha scritto:

...

On Tue, Jun 25, 2019 at 10:26 AM Stefano Danzi <s.danzi@hawai.it> wrote:

...

Il 25/06/2019 08:27, Yedidyah Bar David ha scritto:

...

On Mon, Jun 24, 2019 at 7:56 PM Stefano Danzi <s.danzi@hawai.it> wrote:

...

I've found that this issue is related to:

https://bugzilla.redhat.com/show_bug.cgi?id=1648190 Are you sure?

That bug is about an old cert, generated by an old version, likely before we fixed bug 1210486 (even though it's not mentioned in above bug). Yes! Malformed "Not Before" date/time in certs

...

...

But i've no idea how fix it....

Il 24/06/2019 18:19, Stefano Danzi ha scritto:

...

I've just upgraded my test environment from ovirt 4.2 to 4.3.4. Was it installed as 4.2, or upgraded? From which first version? I don't remember the first installed version. Maybe 4.0... I always upgraded the original installation.

...

...

...

System has only one host (Centos 7.6.1810) and run a self hosted engine.

After upgrade I'm not able to run vdsmd (and so hosted engine....)

Above the error in log:

journalctl -xe

-- L'unità libvirtd.service ha iniziato la fase di avvio. giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: info : libvirt version: 4.5.0, package: 10.el7_6.12 (CentOS BuildSystem <http://bugs.centos.org>, 2019-06-20-15:01:15, x86-01.bsys. giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: info : hostname: ovirt01.hawai.lan giu 24 18:09:17 ovirt01.hawai.lan libvirtd[8176]: 2019-06-24 16:09:17.006+0000: 8176: error : virNetTLSContextLoadCertFromFile:513 : Unable to import server certificate /etc/pki/vdsm/certs/vdsmcert.pem Did you check this file? Does it exist?

ls -l /etc/pki/vdsm/certs/vdsmcert.pem

Can vdsm user read it?

su - vdsm -s /bin/bash -c 'cat /etc/pki/vdsm/certs/vdsmcert.pem > /dev/null'

Please check/share output of:

openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text

Thanks and best regards, vdsm can read vdsmcert. The problem is "Not Before" date:

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text' Certificate: Data: Version: 3 (0x2) Serial Number: 4102 (0x1006) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=hawai.lan, CN=ovirtbk-sheng.hawai.lan.63272 Validity Not Before: Feb 4 08:36:07 2015 Not After : Feb 4 08:36:07 2020 GMT [CUT]

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text' Certificate: Data: Version: 3 (0x2) Serial Number: 4096 (0x1000) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=hawai.lan, CN=ovirtbk-sheng.hawai.lan.63272 Validity Not Before: Feb 4 00:06:25 2015 Not After : Feb 2 00:06:25 2025 GMT

OK :-(

So it will be rather difficult to fix.

You should have been prompted by engine-setup long ago to renew PKI, weren't you? And when you did, didn't you have to reinstall (or Re- Enroll Certificates, in later versions) all hosts?

I don't remember to ever seen a question about this during engine-setup, but it could be. In /etc/pki/vdsm/certs/ I can see an old cert and ca with subjet:

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/cacert.pem.20150205093608 -text' Certificate: Data: Version: 3 (0x2) Serial Number: 1423056193 (0x54d21d41) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=VDSM Certificate Authority Validity Not Before: Feb 4 13:23:13 2015 GMT Not After : Feb 4 13:23:13 2016 GMT Subject: CN=VDSM Certificate Authority Subject Public Key Info:

[CUT]

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem.20150205093609 -text' Certificate: Data: Version: 3 (0x2) Serial Number: 1423056193 (0x54d21d41) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=VDSM Certificate Authority Validity Not Before: Feb 4 13:23:13 2015 GMT Not After : Feb 4 13:23:13 2016 GMT Subject: CN=ovirt01.hawai.lan, O=VDSM Certificate Subject Public Key Info: Public Key Algorithm: rsaEncryption

I think that was certs made during first hosted engine installation. Could it work if I manually create certs like this? Just to start libvirtd, vdsm and hosted-engine.

I think it's worth a try. Just create a self-signed CA, a keypair signed by it, and place them correctly, should work. The engine won't be able to talk with the host, but you can then more easily reinstall/re-enroll-certs. Good luck, -- Didi

Il 25/06/2019 14:26, Stefano Danzi ha scritto:

...

...

I don't remember to ever seen a question about this during engine-setup, but it could be. In /etc/pki/vdsm/certs/ I can see an old cert and ca with subjet:

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/cacert.pem.20150205093608 -text' Certificate:       Data:           Version: 3 (0x2)           Serial Number: 1423056193 (0x54d21d41)       Signature Algorithm: sha256WithRSAEncryption           Issuer: CN=VDSM Certificate Authority           Validity               Not Before: Feb  4 13:23:13 2015 GMT               Not After : Feb  4 13:23:13 2016 GMT           Subject: CN=VDSM Certificate Authority           Subject Public Key Info:

[CUT]

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem.20150205093609 -text' Certificate:       Data:           Version: 3 (0x2)           Serial Number: 1423056193 (0x54d21d41)       Signature Algorithm: sha256WithRSAEncryption           Issuer: CN=VDSM Certificate Authority           Validity               Not Before: Feb  4 13:23:13 2015 GMT               Not After : Feb  4 13:23:13 2016 GMT           Subject: CN=ovirt01.hawai.lan, O=VDSM Certificate           Subject Public Key Info:               Public Key Algorithm: rsaEncryption

I think that was certs made during first hosted engine installation. Could it work if I manually create certs like this? Just to start libvirtd, vdsm and hosted-engine. I think it's worth a try. Just create a self-signed CA, a keypair signed by it, and place them correctly, should work.

The engine won't be able to talk with the host, but you can then more easily reinstall/re-enroll-certs.

Good luck, This workaround works! I have hosted engine running!

So I have to find how reinstall/re-enroll-certs on host. From engine UI host status is "NonResponsive" and I can't do nothing.... _______________________________________________

Status: now Host status is "Unassiged".  Engine can't reach host for "General SSLEngine problem" and It's ok because certs are "home made". I can't switch host to maintenance because it's not operational. I can't enroll certificate because is not in maintenance status. hou I can enroll host cert manually?

Il 26/06/2019 11:57, Yedidyah Bar David ha scritto:

On Tue, Jun 25, 2019 at 8:37 PM Stefano Danzi <s.danzi@hawai.it> wrote:

...

Il 25/06/2019 14:26, Stefano Danzi ha scritto:

...

...

...

I don't remember to ever seen a question about this during engine-setup, but it could be. In /etc/pki/vdsm/certs/ I can see an old cert and ca with subjet:

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/cacert.pem.20150205093608 -text' Certificate: Data: Version: 3 (0x2) Serial Number: 1423056193 (0x54d21d41) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=VDSM Certificate Authority Validity Not Before: Feb 4 13:23:13 2015 GMT Not After : Feb 4 13:23:13 2016 GMT Subject: CN=VDSM Certificate Authority Subject Public Key Info:

[CUT]

[root@ovirt01 ~]# su - vdsm -s /bin/bash -c 'openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem.20150205093609 -text' Certificate: Data: Version: 3 (0x2) Serial Number: 1423056193 (0x54d21d41) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=VDSM Certificate Authority Validity Not Before: Feb 4 13:23:13 2015 GMT Not After : Feb 4 13:23:13 2016 GMT Subject: CN=ovirt01.hawai.lan, O=VDSM Certificate Subject Public Key Info: Public Key Algorithm: rsaEncryption

I think that was certs made during first hosted engine installation. Could it work if I manually create certs like this? Just to start libvirtd, vdsm and hosted-engine. I think it's worth a try. Just create a self-signed CA, a keypair signed by it, and place them correctly, should work.

The engine won't be able to talk with the host, but you can then more easily reinstall/re-enroll-certs.

Good luck, This workaround works! I have hosted engine running!

So I have to find how reinstall/re-enroll-certs on host. From engine UI host status is "NonResponsive" and I can't do nothing.... _______________________________________________ Status:

now Host status is "Unassiged". Engine can't reach host for "General SSLEngine problem" and It's ok because certs are "home made". I can't switch host to maintenance because it's not operational. I can't enroll certificate because is not in maintenance status. You can try to remove it. I think we do not support "force-remove" despite being asked about this occasionally, because generally-speaking, this is very unsafe. If you insist, you can try using the sql function DeleteVds to delete it from the database.

...

hou I can enroll host cert manually? You can try following what I wrote in "2. Try to manually fix" before. Create a CSR on the host (with whatever private key you want), copy it to engine, pki-enroll-request, copy the cert to host.

Good luck and best regards,

I've just solved using pki-enroll-request as you told me. Thanks!! This upgrade was very very hard!!