On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler@redhat.com> wrote:

On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech@gmail.com> wrote:

Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:

names="ovirt-provider-ovn"

subject="$(\
    openssl x509 \
    -in /etc/pki/ovirt-engine/certs/apache.cer \
    -noout \
    -subject | \
        sed \
            's;subject= \(.*\);\1;'
  )"
 
. /usr/share/ovirt-engine/bin/engine-prolog.sh

for name in $names; do
    /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \
        --name="${name}" \
        --password=mypass \
        --subject="${subject}" \
        --keep-key \
        --san=DNS:"${ENGINE_FQDN}"
done

Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other.

I had previously added such logical switched at engine by running:

ovn-nbctl ls-add ovn-net0

ovn-nbctl ls-add ovn-net1

etc

Not related: Please use ovirt-provider-ovn to create and manage ovn entities.

 

Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see:

reconnect|WARN|unix#45: connection dropped (Connection reset by peer)

/var/log/openvswitch/ovn-controller.log might contain the reason.

 

Also systemctl status ovirt-provider-ovn.service at engine shows:

/usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...

Looks not good, do tou know which connection this warning referes to?

 

I have restarted at engine both engine and ovn services:

systemctl restart ovirt-engine

systemctl status ovirt-provider-ovn.service

I have also restarted the relevant service at each host:

systemctl restart ovn-controller.service

When running at host the following it stucks and does not give any output:

ovn-sbctl show

This is expected, the ovn southbound and northbound db exists only on the ovn-central, which is places on the same machine as oVirt Engine.

Only the ovn-controller, which controls openvswitch, and openvswitch, which is implementing the data plane, is placed on the ovn-chassis / oVirt host.

 

I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:

keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore

This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn.

 

At this same cluster, I had previously changed the domain name of each host and engine using the rename tool.

And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.

It seems that OVN is not happy with the status of certificates.

When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.

This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn. The prompt to trust the certificate might be redundant. If you get the green confirmation, oVirt Engine is happy and the certificate of the REST API of ovirt-provider-ovn is fine.

 

Is there anything else that can be done to fix OVN functionality?

Please try to understand what is wrong in the connection between ovn-controller and ovn south bound db.

/var/log/openvswitch/ovn-controller.log should be helpful and might contain the reason.

Will run the steps again to see. Do you think I need to take additional steps when fixing the OVN certs issue due to domain change that this cluster has undergone?

 

Thanx

Alex

On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:

Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate.

DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.

On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:

Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf.

I will repeat the test to verify that all services are functional following this process.

On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:

Hi all,

I am trying to replace the ovirt certificate at ovirt 4.3 following this:

https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/appe-red_hat_enterprise_virtualization_and_ssl

I am doing the following:

I have engine FQDN: manager.lab.local

1. Create root CA private key:
openssl genrsa -des3 -out root.key 2048

2. Generate root certificate: (enter passphrase of root key)
openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem

cp root.pem /tmp

3. Create key and CSR for engine:
openssl genrsa -out manager.lab.local.key 2048
openssl req -new -out manager.lab.local.csr -key manager.lab.local.key

4. Generate a certificate for engine and sign with the root CA key:

openssl x509 -req -in manager.lab.local.csr \
-CA root.pem \
-CAkey root.key \
-CAcreateserial \
-out manager.lab.local.crt \
-days 3650 \
-sha256 \
-extensions v3_req

5. Verify the trust chain and check the certificate details:
openssl verify -CAfile root.pem manager.lab.local.crt
openssl x509 -text -noout -in  manager.lab.local.crt  | head -15

6. Generate a P12 container: (with empty password)
openssl pkcs12 -export -out /tmp/apache.p12 \
-inkey manager.lab.local.key \
-in manager.lab.local.crt

8. Export key and cert:
openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key
openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer

From the above steps we should have the following:

    /tmp/root.pem
    /tmp/apache.p12
    /tmp/apache.key
    /tmp/apache.cer

9. Place the certificates:
hosted-engine --set-maintenance --mode=global

cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck
cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12
cp /tmp/root.pem /etc/pki/ca-trust/source/anchors
update-ca-trust
rm /etc/pki/ovirt-engine/apache-ca.pem
cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem

Backup existing key and cert:
cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck
cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck
cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass
cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer
chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass
chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass
systemctl restart httpd.service

10. Create a new trust store configuration file:
vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf

ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts"
ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""

11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf :
vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf

SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer
SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass

12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf:
vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf

# Key file for SSL connections
ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass
# Certificate file for SSL connections
ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer

13. Import the certificate at system-wide  java trust store

update-ca-trust extract
keytool -list -alias ovirt -keystore /etc/pki/java/cacerts

14. Restart services:
systemctl restart httpd.service
systemctl restart ovirt-provider-ovn.service
systemctl restart ovirt-imageio-proxy
systemctl restart ovirt-websocket-proxy
systemctl restart ovirt-engine.service

Following the above I get at engine GUI:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists.

I have tried several other suggestions from similar issues reported at this list without any luck.

I have run out of ideas. Am I missing anything?

Thanx for any suggestions.

Alex

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOAEGVVTOLGLBFKFLGUM6/