[ovirt-users] Re: Replacing ovirt certificates issue

On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech(a)gmail.com&gt; wrote: > >> Following the above, I was seeing that OVN provider connectivity test >> was failing due to some certificate issue and had to do the following to >> fix it: >> >> names="ovirt-provider-ovn" >> >> subject="$(\ >> openssl x509 \ >> -in /etc/pki/ovirt-engine/certs/apache.cer \ >> -noout \ >> -subject | \ >> sed \ >> 's;subject= \(.*\);\1;' >> )" >> >> . /usr/share/ovirt-engine/bin/engine-prolog.sh >> >> for name in $names; do >> /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ >> --name="${name}" \ >> --password=mypass \ >> --subject="${subject}" \ >> --keep-key \ >> --san=DNS:"${ENGINE_FQDN}" >> done >> >> Having fixed the above, when trying to connect two VMs on some OVN >> logical switches it seems they are not able to reach each other. >> I had previously added such logical switched at engine by running: >> >> ovn-nbctl ls-add ovn-net0 >> ovn-nbctl ls-add ovn-net1 >> etc >> >> > Not related: Please use ovirt-provider-ovn to create and manage ovn > entities. > > >> Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I >> see: >> reconnect|WARN|unix#45: connection dropped (Connection reset by peer) >> >> > /var/log/openvswitch/ovn-controller.log might contain the reason. > > >> Also systemctl status ovirt-provider-ovn.service at engine shows: >> /usr/lib/python2.7/site-packages/urllib3/connection.py:344: >> SubjectAltNameWarning:... >> >> > Looks not good, do tou know which connection this warning referes to? > > >> I have restarted at engine both engine and ovn services: >> systemctl restart ovirt-engine >> systemctl status ovirt-provider-ovn.service >> >> I have also restarted the relevant service at each host: >> systemctl restart ovn-controller.service >> >> When running at host the following it stucks and does not give any >> output: >> ovn-sbctl show >> >> > This is expected, the ovn southbound and northbound db exists only on the > ovn-central, which is places on the same machine as oVirt Engine. > Only the ovn-controller, which controls openvswitch, and openvswitch, > which is implementing the data plane, is placed on the ovn-chassis / oVirt > host. > > >> I see that the certificate is imported at key-store as it has the same >> fingerprint with the previous root CA: >> >> keytool -list -alias ovirt-provider-ovn -keystore >> /var/lib/ovirt-engine/external_truststore >> >> > This is only relevant for the connection from oVirt Engine to > ovirt-provider-ovn. > > >> At this same cluster, I had previously changed the domain name of each >> host and engine using the rename tool. >> And now replaced the certificates as per previous described so as to fix >> the imageio cert issue and ovn issue. >> >> It seems that OVN is not happy with the status of certificates. >> When testing connection at engine GUI i get a prompt to trust the cert, >> and when pressing ok i get a green confirmation of successful connection. >> >> > This is only relevant for the connection from oVirt Engine to > ovirt-provider-ovn. The prompt to trust the certificate might be redundant. > If you get the green confirmation, oVirt Engine is happy and the > certificate of the REST API of ovirt-provider-ovn is fine. > > >> Is there anything else that can be done to fix OVN functionality? >> > > Please try to understand what is wrong in the connection between > ovn-controller and ovn south bound db. > /var/log/openvswitch/ovn-controller.log should be helpful and might > contain the reason. > Will run the steps again to see. Do you think I need to take additional steps when fixing the OVN certs issue due to domain change that this cluster has undergone?

> > >> Thanx >> Alex >> >> >> >> >> >> On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech(a)gmail.com&gt; wrote: >> >>> Seems that all services (imageio, ovn, web socket) are fine after >>> following the above and importing the new self signed CA certificate. >>> DId run also engine-setup as I was trying to fix the imageio cert >>> issue, though seems that that was only fixed after importing the CA cert at >>> browser and engine-setup might not be needed. >>> >>> On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech(a)gmail.com&gt; wrote: >>> >>>> Seems I had a typo at >>>> /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. >>>> I will repeat the test to verify that all services are functional >>>> following this process. >>>> >>>> On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I am trying to replace the ovirt certificate at ovirt 4.3 following >>>>> this: >>>>> >>>>> >>>>> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/... >>>>> >>>>> I am doing the following: >>>>> I have engine FQDN: manager.lab.local >>>>> >>>>> 1. Create root CA private key: >>>>> openssl genrsa -des3 -out root.key 2048 >>>>> >>>>> 2. Generate root certificate: (enter passphrase of root key) >>>>> openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out >>>>> root.pem >>>>> cp root.pem /tmp >>>>> >>>>> 3. Create key and CSR for engine: >>>>> openssl genrsa -out manager.lab.local.key 2048 >>>>> openssl req -new -out manager.lab.local.csr -key manager.lab.local.key >>>>> >>>>> 4. Generate a certificate for engine and sign with the root CA key: >>>>> >>>>> openssl x509 -req -in manager.lab.local.csr \ >>>>> -CA root.pem \ >>>>> -CAkey root.key \ >>>>> -CAcreateserial \ >>>>> -out manager.lab.local.crt \ >>>>> -days 3650 \ >>>>> -sha256 \ >>>>> -extensions v3_req >>>>> >>>>> 5. Verify the trust chain and check the certificate details: >>>>> openssl verify -CAfile root.pem manager.lab.local.crt >>>>> openssl x509 -text -noout -in manager.lab.local.crt | head -15 >>>>> >>>>> 6. Generate a P12 container: (with empty password) >>>>> openssl pkcs12 -export -out /tmp/apache.p12 \ >>>>> -inkey manager.lab.local.key \ >>>>> -in manager.lab.local.crt >>>>> >>>>> 8. Export key and cert: >>>>> openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key >>>>> openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer >>>>> >>>>> From the above steps we should have the following: >>>>> >>>>> /tmp/root.pem >>>>> /tmp/apache.p12 >>>>> /tmp/apache.key >>>>> /tmp/apache.cer >>>>> >>>>> 9. Place the certificates: >>>>> hosted-engine --set-maintenance --mode=global >>>>> cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck >>>>> cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 >>>>> cp /tmp/root.pem /etc/pki/ca-trust/source/anchors >>>>> update-ca-trust >>>>> rm /etc/pki/ovirt-engine/apache-ca.pem >>>>> cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem >>>>> >>>>> Backup existing key and cert: >>>>> cp /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>> /etc/pki/ovirt-engine/keys/apache.key.nopass.bck >>>>> cp /etc/pki/ovirt-engine/certs/apache.cer >>>>> /etc/pki/ovirt-engine/certs/apache.cer.bck >>>>> cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>> cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer >>>>> chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>> chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>> systemctl restart httpd.service >>>>> >>>>> 10. Create a new trust store configuration file: >>>>> vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf >>>>> >>>>> ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" >>>>> ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" >>>>> >>>>> 11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf >>>>> : >>>>> vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf >>>>> >>>>> SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer >>>>> SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass >>>>> >>>>> 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: >>>>> vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf >>>>> >>>>> # Key file for SSL connections >>>>> ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>> # Certificate file for SSL connections >>>>> ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer >>>>> >>>>> 13. Import the certificate at system-wide java trust store >>>>> >>>>> update-ca-trust extract >>>>> keytool -list -alias ovirt -keystore /etc/pki/java/cacerts >>>>> >>>>> 14. Restart services: >>>>> systemctl restart httpd.service >>>>> systemctl restart ovirt-provider-ovn.service >>>>> systemctl restart ovirt-imageio-proxy >>>>> systemctl restart ovirt-websocket-proxy >>>>> systemctl restart ovirt-engine.service >>>>> >>>>> Following the above I get at engine GUI: >>>>> >>>>> sun.security.validator.ValidatorException: PKIX path building failed: >>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find >>>>> valid certification path to requested target >>>>> >>>>> I have tried also to run engine-setup in case it could fix anything >>>>> (it renewed the cert due to missing subjectAltName), and the above error >>>>> still persists. >>>>> I have tried several other suggestions from similar issues reported >>>>> at this list without any luck. >>>>> I have run out of ideas. Am I missing anything? >>>>> Thanx for any suggestions. >>>>> Alex >>>>> >>>> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZB... >> >