Replacing ovirt certificates issue
Hi all, I am trying to replace the ovirt certificate at ovirt 4.3 following this: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm... I am doing the following: I have engine FQDN: manager.lab.local 1. Create root CA private key: openssl genrsa -des3 -out root.key 2048 2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp 3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key 4. Generate a certificate for engine and sign with the root CA key: openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req 5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15 6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt 8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer From the above steps we should have the following: /tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer 9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service 10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" 11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf # Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer 13. Import the certificate at system-wide java trust store update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts 14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service Following the above I get at engine GUI: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process. On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed. On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it: names="ovirt-provider-ovn" subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )" . /usr/share/ovirt-engine/bin/engine-prolog.sh for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running: ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer) Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:... I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service I have also restarted the relevant service at each host: systemctl restart ovn-controller.service When running at host the following it stucks and does not give any output: ovn-sbctl show I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA: keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore At this same cluster, I had previously changed the domain name of each host and engine using the rename tool. And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue. It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection. Is there anything else that can be done to fix OVN functionality? Thanx Alex On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
On Fri, Nov 20, 2020 at 1:37 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
Is this on the same systems of "[ovirt-users] Fix corrupt self-hosted engine", or unrelated?
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool.
After that, did ovn still work well?
And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
Is there anything else that can be done to fix OVN functionality?
No idea, adding Dominik. Best regards,
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
-- Didi
On Sun, Nov 22, 2020 at 9:11 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Nov 20, 2020 at 1:37 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
Is this on the same systems of "[ovirt-users] Fix corrupt self-hosted engine", or unrelated?
Yes, it is the same one.
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool.
After that, did ovn still work well?
No. When clicking at the "test" button, I am prompted to accept self signed cert and then I receive a green check confirmation that all is good but when attaching two guest net ports on same OVN logicat switch the VMs are not able to communicate and was observing some SSL issues at OVN logs referring to subjAltName issues.
And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
Is there anything else that can be done to fix OVN functionality?
No idea, adding Dominik.
The cluster I am trying to fix was renamed at a previous point. I had to change the domain where the hosts and the engine belonged. Did that, with few hurdles (using the rename tool available at 4.3). That procedure left out the omageio and OVN certs which I want to resolve now as I need OVN to be able to setup test environment. I checked also https://bugzilla.redhat.com/show_bug.cgi?id=1501798, and proceeded to generate new certs for ovn-ndb and ovn-sdb. Following this, and after running engine-setup I observed that the host had to be re-installed as per web UI prompt. I tried to, but host installation was not being completed due to "General SSLEngine problem". Tried also to renew the vdsm certs at the host, using same vdsm key and the new CA cert, but without luck. At that point I was receiving "Get Host Capabilities failed: Received fatal alert: unknown_ca" and the other host also became no-responsive. Generally I tried several things (even renewing at engine the engine.cer cert using the engine_id_rsa key at engine) but the host was refusing to install. So it seems that when a domain change is involved in the scenario, I have to include some additional steps to make the setup happy. I had to roll-back the pki at its previous state so as to recover one host and re-install the other, as the site is production. I have read that this the engine rename tool at 4.4 has several fixes and was thinking maybe I can use it at 4.3? Is it sth else that can be done to fix the vdsm cert issues or change the certs in a way to take into account the new domain name? Best regards,
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
-- Didi
On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Not related: Please use ovirt-provider-ovn to create and manage ovn entities.
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
/var/log/openvswitch/ovn-controller.log might contain the reason.
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
Looks not good, do tou know which connection this warning referes to?
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
This is expected, the ovn southbound and northbound db exists only on the ovn-central, which is places on the same machine as oVirt Engine. Only the ovn-controller, which controls openvswitch, and openvswitch, which is implementing the data plane, is placed on the ovn-chassis / oVirt host.
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn.
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool. And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn. The prompt to trust the certificate might be redundant. If you get the green confirmation, oVirt Engine is happy and the certificate of the REST API of ovirt-provider-ovn is fine.
Is there anything else that can be done to fix OVN functionality?
Please try to understand what is wrong in the connection between ovn-controller and ovn south bound db. /var/log/openvswitch/ovn-controller.log should be helpful and might contain the reason.
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler@redhat.com> wrote:
On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Not related: Please use ovirt-provider-ovn to create and manage ovn entities.
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
/var/log/openvswitch/ovn-controller.log might contain the reason.
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
Looks not good, do tou know which connection this warning referes to?
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
This is expected, the ovn southbound and northbound db exists only on the ovn-central, which is places on the same machine as oVirt Engine. Only the ovn-controller, which controls openvswitch, and openvswitch, which is implementing the data plane, is placed on the ovn-chassis / oVirt host.
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn.
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool. And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn. The prompt to trust the certificate might be redundant. If you get the green confirmation, oVirt Engine is happy and the certificate of the REST API of ovirt-provider-ovn is fine.
Is there anything else that can be done to fix OVN functionality?
Please try to understand what is wrong in the connection between ovn-controller and ovn south bound db. /var/log/openvswitch/ovn-controller.log should be helpful and might contain the reason.
Will run the steps again to see. Do you think I need to take additional steps when fixing the OVN certs issue due to domain change that this cluster has undergone?
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
On Mon, Nov 23, 2020 at 9:42 AM Alex K <rightkicktech@gmail.com> wrote:
On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler@redhat.com> wrote:
On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Not related: Please use ovirt-provider-ovn to create and manage ovn entities.
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
/var/log/openvswitch/ovn-controller.log might contain the reason.
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
Looks not good, do tou know which connection this warning referes to?
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
This is expected, the ovn southbound and northbound db exists only on the ovn-central, which is places on the same machine as oVirt Engine. Only the ovn-controller, which controls openvswitch, and openvswitch, which is implementing the data plane, is placed on the ovn-chassis / oVirt host.
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn.
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool. And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn. The prompt to trust the certificate might be redundant. If you get the green confirmation, oVirt Engine is happy and the certificate of the REST API of ovirt-provider-ovn is fine.
Is there anything else that can be done to fix OVN functionality?
Please try to understand what is wrong in the connection between ovn-controller and ovn south bound db. /var/log/openvswitch/ovn-controller.log should be helpful and might contain the reason.
Will run the steps again to see. Do you think I need to take additional steps when fixing the OVN certs issue due to domain change that this cluster has undergone?
This time was not able to make OVN provider succeed at the test connection with the new certs. Restored the pki to its previous state, using the previous CA. Now, I do not see any errors for the last hour. Created also from GUI a logical switch named ovn-switch-1. Attached two VMs to it and was not able to confirm ping between VMs. At engine I see both the MAC addresses of each guest VM. [root@engine ~]# ovn-nbctl show switch ae4e03eb-e097-4629-a7bd-3272eee65599 (ovirt-ovn-switch-1-bd04ad69-11a9-46d0-b571-f7dee62dfb7c) port c4e4098b-764f-4696-8506-ccf46a535fd2 addresses: ["00:1a:4a:16:02:59"] port 14b8fb48-5ab1-4b10-93e1-d4e1fab17b51 addresses: ["00:1a:4a:16:02:5e"] and the two hosts: [root@engine ~]# ovn-sbctl show Chassis "580a335e-f55f-4947-95d5-e90690b05125" hostname: "v1" Encap geneve ip: "10.10.10.12" options: {csum="true"} Port_Binding "14b8fb48-5ab1-4b10-93e1-d4e1fab17b51" Port_Binding "c4e4098b-764f-4696-8506-ccf46a535fd2" Chassis "872e9ea7-fe6e-455f-8645-a2d6159c7552" hostname: "v0" Encap geneve ip: "10.10.10.11" options: {csum="true"} The status of OVN provider seem fine at engine: [root@engine ~]# systemctl status ovirt-provider-ovn.service ● ovirt-provider-ovn.service - oVirt OVN provider Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2020-11-23 16:40:33 EET; 10min ago Main PID: 25293 (python2) Tasks: 4 CGroup: /system.slice/ovirt-provider-ovn.service └─25293 /usr/bin/python2 /usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py Nov 23 16:40:33 engine.mtis.tech systemd[1]: Started oVirt OVN provider. Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - [23/Nov/2020 16:44:26] "POST /v2.0//tokens HTTP/1.1" 200 - Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - [23/Nov/2020 16:44:26] "GET /v2.0/networks HTTP/1.1" 200 - Also the status of OVN controller at each host seem fine: [root@v0 ~]# systemctl status ovn-controller.service ● ovn-controller.service - OVN controller daemon Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2020-11-22 11:43:17 EET; 1 day 5h ago Process: 11492 ExecStop=/usr/share/openvswitch/scripts/ovn-ctl stop_controller (code=exited, status=0/SUCCESS) Process: 11509 ExecStart=/usr/share/openvswitch/scripts/ovn-ctl --no-monitor start_controller $OVN_CONTROLLER_OPTS (code=exited, status=0/SUCCESS) Main PID: 11539 (ovn-controller) Tasks: 4 CGroup: /system.slice/ovn-controller.service └─11539 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --private-key=/etc/pki... Nov 22 11:43:17 v0.mtis.tech systemd[1]: Starting OVN controller daemon... Nov 22 11:43:17 v0.mtis.tech ovn-ctl[11509]: Starting ovn-controller [ OK ] Nov 22 11:43:17 v0.mtis.tech systemd[1]: Started OVN controller daemon. What could be the reason of traffic not being forwarded through the logical switch?
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
Hi all,
I am trying to replace the ovirt certificate at ovirt 4.3 following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm...
I am doing the following: I have engine FQDN: manager.lab.local
1. Create root CA private key: openssl genrsa -des3 -out root.key 2048
2. Generate root certificate: (enter passphrase of root key) openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out root.pem cp root.pem /tmp
3. Create key and CSR for engine: openssl genrsa -out manager.lab.local.key 2048 openssl req -new -out manager.lab.local.csr -key manager.lab.local.key
4. Generate a certificate for engine and sign with the root CA key:
openssl x509 -req -in manager.lab.local.csr \ -CA root.pem \ -CAkey root.key \ -CAcreateserial \ -out manager.lab.local.crt \ -days 3650 \ -sha256 \ -extensions v3_req
5. Verify the trust chain and check the certificate details: openssl verify -CAfile root.pem manager.lab.local.crt openssl x509 -text -noout -in manager.lab.local.crt | head -15
6. Generate a P12 container: (with empty password) openssl pkcs12 -export -out /tmp/apache.p12 \ -inkey manager.lab.local.key \ -in manager.lab.local.crt
8. Export key and cert: openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer
From the above steps we should have the following:
/tmp/root.pem /tmp/apache.p12 /tmp/apache.key /tmp/apache.cer
9. Place the certificates: hosted-engine --set-maintenance --mode=global cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 cp /tmp/root.pem /etc/pki/ca-trust/source/anchors update-ca-trust rm /etc/pki/ovirt-engine/apache-ca.pem cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem
Backup existing key and cert: cp /etc/pki/ovirt-engine/keys/apache.key.nopass /etc/pki/ovirt-engine/keys/apache.key.nopass.bck cp /etc/pki/ovirt-engine/certs/apache.cer /etc/pki/ovirt-engine/certs/apache.cer.bck cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass systemctl restart httpd.service
10. Create a new trust store configuration file: vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
11. Edit /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf
SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass
12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf
# Key file for SSL connections ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass # Certificate file for SSL connections ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer
13. Import the certificate at system-wide java trust store
update-ca-trust extract keytool -list -alias ovirt -keystore /etc/pki/java/cacerts
14. Restart services: systemctl restart httpd.service systemctl restart ovirt-provider-ovn.service systemctl restart ovirt-imageio-proxy systemctl restart ovirt-websocket-proxy systemctl restart ovirt-engine.service
Following the above I get at engine GUI:
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I have tried also to run engine-setup in case it could fix anything (it renewed the cert due to missing subjectAltName), and the above error still persists. I have tried several other suggestions from similar issues reported at this list without any luck. I have run out of ideas. Am I missing anything? Thanx for any suggestions. Alex
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
On Mon, Nov 23, 2020 at 4:54 PM Alex K <rightkicktech@gmail.com> wrote:
On Mon, Nov 23, 2020 at 9:42 AM Alex K <rightkicktech@gmail.com> wrote:
On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler@redhat.com> wrote:
On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Not related: Please use ovirt-provider-ovn to create and manage ovn entities.
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
/var/log/openvswitch/ovn-controller.log might contain the reason.
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
Looks not good, do tou know which connection this warning referes to?
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
This is expected, the ovn southbound and northbound db exists only on the ovn-central, which is places on the same machine as oVirt Engine. Only the ovn-controller, which controls openvswitch, and openvswitch, which is implementing the data plane, is placed on the ovn-chassis / oVirt host.
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn.
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool. And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn. The prompt to trust the certificate might be redundant. If you get the green confirmation, oVirt Engine is happy and the certificate of the REST API of ovirt-provider-ovn is fine.
Is there anything else that can be done to fix OVN functionality?
Please try to understand what is wrong in the connection between ovn-controller and ovn south bound db. /var/log/openvswitch/ovn-controller.log should be helpful and might contain the reason.
Will run the steps again to see. Do you think I need to take additional steps when fixing the OVN certs issue due to domain change that this cluster has undergone?
This time was not able to make OVN provider succeed at the test connection with the new certs. Restored the pki to its previous state, using the previous CA. Now, I do not see any errors for the last hour. Created also from GUI a logical switch named ovn-switch-1. Attached two VMs to it and was not able to confirm ping between VMs.
At engine I see both the MAC addresses of each guest VM. [root@engine ~]# ovn-nbctl show switch ae4e03eb-e097-4629-a7bd-3272eee65599 (ovirt-ovn-switch-1-bd04ad69-11a9-46d0-b571-f7dee62dfb7c) port c4e4098b-764f-4696-8506-ccf46a535fd2 addresses: ["00:1a:4a:16:02:59"] port 14b8fb48-5ab1-4b10-93e1-d4e1fab17b51 addresses: ["00:1a:4a:16:02:5e"]
and the two hosts: [root@engine ~]# ovn-sbctl show Chassis "580a335e-f55f-4947-95d5-e90690b05125" hostname: "v1" Encap geneve ip: "10.10.10.12" options: {csum="true"} Port_Binding "14b8fb48-5ab1-4b10-93e1-d4e1fab17b51" Port_Binding "c4e4098b-764f-4696-8506-ccf46a535fd2" Chassis "872e9ea7-fe6e-455f-8645-a2d6159c7552" hostname: "v0" Encap geneve ip: "10.10.10.11" options: {csum="true"}
The status of OVN provider seem fine at engine: [root@engine ~]# systemctl status ovirt-provider-ovn.service ● ovirt-provider-ovn.service - oVirt OVN provider Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2020-11-23 16:40:33 EET; 10min ago Main PID: 25293 (python2) Tasks: 4 CGroup: /system.slice/ovirt-provider-ovn.service └─25293 /usr/bin/python2 /usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py
Nov 23 16:40:33 engine.mtis.tech systemd[1]: Started oVirt OVN provider. Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - [23/Nov/2020 16:44:26] "POST /v2.0//tokens HTTP/1.1" 200 - Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - [23/Nov/2020 16:44:26] "GET /v2.0/networks HTTP/1.1" 200 -
Also the status of OVN controller at each host seem fine: [root@v0 ~]# systemctl status ovn-controller.service ● ovn-controller.service - OVN controller daemon Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2020-11-22 11:43:17 EET; 1 day 5h ago Process: 11492 ExecStop=/usr/share/openvswitch/scripts/ovn-ctl stop_controller (code=exited, status=0/SUCCESS) Process: 11509 ExecStart=/usr/share/openvswitch/scripts/ovn-ctl --no-monitor start_controller $OVN_CONTROLLER_OPTS (code=exited, status=0/SUCCESS) Main PID: 11539 (ovn-controller) Tasks: 4 CGroup: /system.slice/ovn-controller.service └─11539 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --private-key=/etc/pki...
Nov 22 11:43:17 v0.mtis.tech systemd[1]: Starting OVN controller daemon... Nov 22 11:43:17 v0.mtis.tech ovn-ctl[11509]: Starting ovn-controller [ OK ] Nov 22 11:43:17 v0.mtis.tech systemd[1]: Started OVN controller daemon.
What could be the reason of traffic not being forwarded through the logical switch?
I noticed that there is a network port security setting when defining the network at the external provider, which is enabled by default. [image: image.png] Setting this to disabled made the VMs attached to the same network to be able to communicate through it. It seems that when enabled this does drop all traffic by default if not specifically configured. I am wondering where I could configure a security group/policy to allow such traffic.
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
Seems I had a typo at /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. I will repeat the test to verify that all services are functional following this process.
On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> wrote:
> Hi all, > > I am trying to replace the ovirt certificate at ovirt 4.3 following > this: > > > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm... > > I am doing the following: > I have engine FQDN: manager.lab.local > > 1. Create root CA private key: > openssl genrsa -des3 -out root.key 2048 > > 2. Generate root certificate: (enter passphrase of root key) > openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out > root.pem > cp root.pem /tmp > > 3. Create key and CSR for engine: > openssl genrsa -out manager.lab.local.key 2048 > openssl req -new -out manager.lab.local.csr -key > manager.lab.local.key > > 4. Generate a certificate for engine and sign with the root CA key: > > openssl x509 -req -in manager.lab.local.csr \ > -CA root.pem \ > -CAkey root.key \ > -CAcreateserial \ > -out manager.lab.local.crt \ > -days 3650 \ > -sha256 \ > -extensions v3_req > > 5. Verify the trust chain and check the certificate details: > openssl verify -CAfile root.pem manager.lab.local.crt > openssl x509 -text -noout -in manager.lab.local.crt | head -15 > > 6. Generate a P12 container: (with empty password) > openssl pkcs12 -export -out /tmp/apache.p12 \ > -inkey manager.lab.local.key \ > -in manager.lab.local.crt > > 8. Export key and cert: > openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key > openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer > > From the above steps we should have the following: > > /tmp/root.pem > /tmp/apache.p12 > /tmp/apache.key > /tmp/apache.cer > > 9. Place the certificates: > hosted-engine --set-maintenance --mode=global > cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck > cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 > cp /tmp/root.pem /etc/pki/ca-trust/source/anchors > update-ca-trust > rm /etc/pki/ovirt-engine/apache-ca.pem > cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem > > Backup existing key and cert: > cp /etc/pki/ovirt-engine/keys/apache.key.nopass > /etc/pki/ovirt-engine/keys/apache.key.nopass.bck > cp /etc/pki/ovirt-engine/certs/apache.cer > /etc/pki/ovirt-engine/certs/apache.cer.bck > cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass > cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer > chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass > chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass > systemctl restart httpd.service > > 10. Create a new trust store configuration file: > vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf > > ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" > ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" > > 11. Edit > /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : > vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf > > SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer > SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass > > 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: > vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf > > # Key file for SSL connections > ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass > # Certificate file for SSL connections > ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer > > 13. Import the certificate at system-wide java trust store > > update-ca-trust extract > keytool -list -alias ovirt -keystore /etc/pki/java/cacerts > > 14. Restart services: > systemctl restart httpd.service > systemctl restart ovirt-provider-ovn.service > systemctl restart ovirt-imageio-proxy > systemctl restart ovirt-websocket-proxy > systemctl restart ovirt-engine.service > > Following the above I get at engine GUI: > > sun.security.validator.ValidatorException: PKIX path building > failed: sun.security.provider.certpath.SunCertPathBuilderException: unable > to find valid certification path to requested target > > I have tried also to run engine-setup in case it could fix anything > (it renewed the cert due to missing subjectAltName), and the above error > still persists. > I have tried several other suggestions from similar issues reported > at this list without any luck. > I have run out of ideas. Am I missing anything? > Thanx for any suggestions. > Alex > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
On Tue, Nov 24, 2020 at 8:45 AM Alex K <rightkicktech@gmail.com> wrote:
On Mon, Nov 23, 2020 at 4:54 PM Alex K <rightkicktech@gmail.com> wrote:
On Mon, Nov 23, 2020 at 9:42 AM Alex K <rightkicktech@gmail.com> wrote:
On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler@redhat.com> wrote:
On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech@gmail.com> wrote:
Following the above, I was seeing that OVN provider connectivity test was failing due to some certificate issue and had to do the following to fix it:
names="ovirt-provider-ovn"
subject="$(\ openssl x509 \ -in /etc/pki/ovirt-engine/certs/apache.cer \ -noout \ -subject | \ sed \ 's;subject= \(.*\);\1;' )"
. /usr/share/ovirt-engine/bin/engine-prolog.sh
for name in $names; do /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ --name="${name}" \ --password=mypass \ --subject="${subject}" \ --keep-key \ --san=DNS:"${ENGINE_FQDN}" done
Having fixed the above, when trying to connect two VMs on some OVN logical switches it seems they are not able to reach each other. I had previously added such logical switched at engine by running:
ovn-nbctl ls-add ovn-net0 ovn-nbctl ls-add ovn-net1 etc
Not related: Please use ovirt-provider-ovn to create and manage ovn entities.
Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I see: reconnect|WARN|unix#45: connection dropped (Connection reset by peer)
/var/log/openvswitch/ovn-controller.log might contain the reason.
Also systemctl status ovirt-provider-ovn.service at engine shows: /usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning:...
Looks not good, do tou know which connection this warning referes to?
I have restarted at engine both engine and ovn services: systemctl restart ovirt-engine systemctl status ovirt-provider-ovn.service
I have also restarted the relevant service at each host: systemctl restart ovn-controller.service
When running at host the following it stucks and does not give any output: ovn-sbctl show
This is expected, the ovn southbound and northbound db exists only on the ovn-central, which is places on the same machine as oVirt Engine. Only the ovn-controller, which controls openvswitch, and openvswitch, which is implementing the data plane, is placed on the ovn-chassis / oVirt host.
I see that the certificate is imported at key-store as it has the same fingerprint with the previous root CA:
keytool -list -alias ovirt-provider-ovn -keystore /var/lib/ovirt-engine/external_truststore
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn.
At this same cluster, I had previously changed the domain name of each host and engine using the rename tool. And now replaced the certificates as per previous described so as to fix the imageio cert issue and ovn issue.
It seems that OVN is not happy with the status of certificates. When testing connection at engine GUI i get a prompt to trust the cert, and when pressing ok i get a green confirmation of successful connection.
This is only relevant for the connection from oVirt Engine to ovirt-provider-ovn. The prompt to trust the certificate might be redundant. If you get the green confirmation, oVirt Engine is happy and the certificate of the REST API of ovirt-provider-ovn is fine.
Is there anything else that can be done to fix OVN functionality?
Please try to understand what is wrong in the connection between ovn-controller and ovn south bound db. /var/log/openvswitch/ovn-controller.log should be helpful and might contain the reason.
Will run the steps again to see. Do you think I need to take additional steps when fixing the OVN certs issue due to domain change that this cluster has undergone?
This time was not able to make OVN provider succeed at the test connection with the new certs. Restored the pki to its previous state, using the previous CA. Now, I do not see any errors for the last hour. Created also from GUI a logical switch named ovn-switch-1. Attached two VMs to it and was not able to confirm ping between VMs.
At engine I see both the MAC addresses of each guest VM. [root@engine ~]# ovn-nbctl show switch ae4e03eb-e097-4629-a7bd-3272eee65599 (ovirt-ovn-switch-1-bd04ad69-11a9-46d0-b571-f7dee62dfb7c) port c4e4098b-764f-4696-8506-ccf46a535fd2 addresses: ["00:1a:4a:16:02:59"] port 14b8fb48-5ab1-4b10-93e1-d4e1fab17b51 addresses: ["00:1a:4a:16:02:5e"]
and the two hosts: [root@engine ~]# ovn-sbctl show Chassis "580a335e-f55f-4947-95d5-e90690b05125" hostname: "v1" Encap geneve ip: "10.10.10.12" options: {csum="true"} Port_Binding "14b8fb48-5ab1-4b10-93e1-d4e1fab17b51" Port_Binding "c4e4098b-764f-4696-8506-ccf46a535fd2" Chassis "872e9ea7-fe6e-455f-8645-a2d6159c7552" hostname: "v0" Encap geneve ip: "10.10.10.11" options: {csum="true"}
The status of OVN provider seem fine at engine: [root@engine ~]# systemctl status ovirt-provider-ovn.service ● ovirt-provider-ovn.service - oVirt OVN provider Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2020-11-23 16:40:33 EET; 10min ago Main PID: 25293 (python2) Tasks: 4 CGroup: /system.slice/ovirt-provider-ovn.service └─25293 /usr/bin/python2 /usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py
Nov 23 16:40:33 engine.mtis.tech systemd[1]: Started oVirt OVN provider. Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - [23/Nov/2020 16:44:26] "POST /v2.0//tokens HTTP/1.1" 200 - Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - [23/Nov/2020 16:44:26] "GET /v2.0/networks HTTP/1.1" 200 -
Also the status of OVN controller at each host seem fine: [root@v0 ~]# systemctl status ovn-controller.service ● ovn-controller.service - OVN controller daemon Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2020-11-22 11:43:17 EET; 1 day 5h ago Process: 11492 ExecStop=/usr/share/openvswitch/scripts/ovn-ctl stop_controller (code=exited, status=0/SUCCESS) Process: 11509 ExecStart=/usr/share/openvswitch/scripts/ovn-ctl --no-monitor start_controller $OVN_CONTROLLER_OPTS (code=exited, status=0/SUCCESS) Main PID: 11539 (ovn-controller) Tasks: 4 CGroup: /system.slice/ovn-controller.service └─11539 ovn-controller unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --private-key=/etc/pki...
Nov 22 11:43:17 v0.mtis.tech systemd[1]: Starting OVN controller daemon... Nov 22 11:43:17 v0.mtis.tech ovn-ctl[11509]: Starting ovn-controller [ OK ] Nov 22 11:43:17 v0.mtis.tech systemd[1]: Started OVN controller daemon.
What could be the reason of traffic not being forwarded through the logical switch?
I noticed that there is a network port security setting when defining the network at the external provider, which is enabled by default.
[image: image.png] Setting this to disabled made the VMs attached to the same network to be able to communicate through it.
Happy to hear.
It seems that when enabled this does drop all traffic by default if not specifically configured. I am wondering where I could configure a security group/policy to allow such traffic.
oVirt implements a subset of the OpenStack Networking API, please report deviant behavior as a bug, including the default behavior without explicit configuration. Please find the details about the network port security in https://github.com/oVirt/ovirt-provider-ovn/blob/master/docs/provider_api_de...
Thanx Alex
On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech@gmail.com> wrote:
Seems that all services (imageio, ovn, web socket) are fine after following the above and importing the new self signed CA certificate. DId run also engine-setup as I was trying to fix the imageio cert issue, though seems that that was only fixed after importing the CA cert at browser and engine-setup might not be needed.
On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech@gmail.com> wrote:
> Seems I had a typo at > /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. > I will repeat the test to verify that all services are functional > following this process. > > On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech@gmail.com> > wrote: > >> Hi all, >> >> I am trying to replace the ovirt certificate at ovirt 4.3 following >> this: >> >> >> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/htm... >> >> I am doing the following: >> I have engine FQDN: manager.lab.local >> >> 1. Create root CA private key: >> openssl genrsa -des3 -out root.key 2048 >> >> 2. Generate root certificate: (enter passphrase of root key) >> openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out >> root.pem >> cp root.pem /tmp >> >> 3. Create key and CSR for engine: >> openssl genrsa -out manager.lab.local.key 2048 >> openssl req -new -out manager.lab.local.csr -key >> manager.lab.local.key >> >> 4. Generate a certificate for engine and sign with the root CA key: >> >> openssl x509 -req -in manager.lab.local.csr \ >> -CA root.pem \ >> -CAkey root.key \ >> -CAcreateserial \ >> -out manager.lab.local.crt \ >> -days 3650 \ >> -sha256 \ >> -extensions v3_req >> >> 5. Verify the trust chain and check the certificate details: >> openssl verify -CAfile root.pem manager.lab.local.crt >> openssl x509 -text -noout -in manager.lab.local.crt | head -15 >> >> 6. Generate a P12 container: (with empty password) >> openssl pkcs12 -export -out /tmp/apache.p12 \ >> -inkey manager.lab.local.key \ >> -in manager.lab.local.crt >> >> 8. Export key and cert: >> openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key >> openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer >> >> From the above steps we should have the following: >> >> /tmp/root.pem >> /tmp/apache.p12 >> /tmp/apache.key >> /tmp/apache.cer >> >> 9. Place the certificates: >> hosted-engine --set-maintenance --mode=global >> cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck >> cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 >> cp /tmp/root.pem /etc/pki/ca-trust/source/anchors >> update-ca-trust >> rm /etc/pki/ovirt-engine/apache-ca.pem >> cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem >> >> Backup existing key and cert: >> cp /etc/pki/ovirt-engine/keys/apache.key.nopass >> /etc/pki/ovirt-engine/keys/apache.key.nopass.bck >> cp /etc/pki/ovirt-engine/certs/apache.cer >> /etc/pki/ovirt-engine/certs/apache.cer.bck >> cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass >> cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer >> chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass >> chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass >> systemctl restart httpd.service >> >> 10. Create a new trust store configuration file: >> vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf >> >> ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" >> ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" >> >> 11. Edit >> /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : >> vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf >> >> SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer >> SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass >> >> 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: >> vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf >> >> # Key file for SSL connections >> ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass >> # Certificate file for SSL connections >> ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer >> >> 13. Import the certificate at system-wide java trust store >> >> update-ca-trust extract >> keytool -list -alias ovirt -keystore /etc/pki/java/cacerts >> >> 14. Restart services: >> systemctl restart httpd.service >> systemctl restart ovirt-provider-ovn.service >> systemctl restart ovirt-imageio-proxy >> systemctl restart ovirt-websocket-proxy >> systemctl restart ovirt-engine.service >> >> Following the above I get at engine GUI: >> >> sun.security.validator.ValidatorException: PKIX path building >> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable >> to find valid certification path to requested target >> >> I have tried also to run engine-setup in case it could fix anything >> (it renewed the cert due to missing subjectAltName), and the above error >> still persists. >> I have tried several other suggestions from similar issues reported >> at this list without any luck. >> I have run out of ideas. Am I missing anything? >> Thanx for any suggestions. >> Alex >> > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZBEOA...
participants (3)
-
Alex K -
Dominik Holler -
Yedidyah Bar David