[ovirt-users] Re: Replacing ovirt certificates issue

On Mon, Nov 23, 2020 at 4:54 PM Alex K <rightkicktech(a)gmail.com&gt; wrote: > > > On Mon, Nov 23, 2020 at 9:42 AM Alex K <rightkicktech(a)gmail.com&gt; wrote: > >> >> >> On Mon, Nov 23, 2020 at 9:35 AM Dominik Holler <dholler(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Fri, Nov 20, 2020 at 12:38 PM Alex K <rightkicktech(a)gmail.com&gt; >>> wrote: >>> >>>> Following the above, I was seeing that OVN provider connectivity test >>>> was failing due to some certificate issue and had to do the following to >>>> fix it: >>>> >>>> names="ovirt-provider-ovn" >>>> >>>> subject="$(\ >>>> openssl x509 \ >>>> -in /etc/pki/ovirt-engine/certs/apache.cer \ >>>> -noout \ >>>> -subject | \ >>>> sed \ >>>> 's;subject= \(.*\);\1;' >>>> )" >>>> >>>> . /usr/share/ovirt-engine/bin/engine-prolog.sh >>>> >>>> for name in $names; do >>>> /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh \ >>>> --name="${name}" \ >>>> --password=mypass \ >>>> --subject="${subject}" \ >>>> --keep-key \ >>>> --san=DNS:"${ENGINE_FQDN}" >>>> done >>>> >>>> Having fixed the above, when trying to connect two VMs on some OVN >>>> logical switches it seems they are not able to reach each other. >>>> I had previously added such logical switched at engine by running: >>>> >>>> ovn-nbctl ls-add ovn-net0 >>>> ovn-nbctl ls-add ovn-net1 >>>> etc >>>> >>>> >>> Not related: Please use ovirt-provider-ovn to create and manage ovn >>> entities. >>> >>> >>>> Checking the logs at the host /var/log/openvswitch/ovsdb-server.log I >>>> see: >>>> reconnect|WARN|unix#45: connection dropped (Connection reset by peer) >>>> >>>> >>> /var/log/openvswitch/ovn-controller.log might contain the reason. >>> >>> >>>> Also systemctl status ovirt-provider-ovn.service at engine shows: >>>> /usr/lib/python2.7/site-packages/urllib3/connection.py:344: >>>> SubjectAltNameWarning:... >>>> >>>> >>> Looks not good, do tou know which connection this warning referes to? >>> >>> >>>> I have restarted at engine both engine and ovn services: >>>> systemctl restart ovirt-engine >>>> systemctl status ovirt-provider-ovn.service >>>> >>>> I have also restarted the relevant service at each host: >>>> systemctl restart ovn-controller.service >>>> >>>> When running at host the following it stucks and does not give any >>>> output: >>>> ovn-sbctl show >>>> >>>> >>> This is expected, the ovn southbound and northbound db exists only on >>> the ovn-central, which is places on the same machine as oVirt Engine. >>> Only the ovn-controller, which controls openvswitch, and openvswitch, >>> which is implementing the data plane, is placed on the ovn-chassis / oVirt >>> host. >>> >>> >>>> I see that the certificate is imported at key-store as it has the same >>>> fingerprint with the previous root CA: >>>> >>>> keytool -list -alias ovirt-provider-ovn -keystore >>>> /var/lib/ovirt-engine/external_truststore >>>> >>>> >>> This is only relevant for the connection from oVirt Engine to >>> ovirt-provider-ovn. >>> >>> >>>> At this same cluster, I had previously changed the domain name of each >>>> host and engine using the rename tool. >>>> And now replaced the certificates as per previous described so as to >>>> fix the imageio cert issue and ovn issue. >>>> >>>> It seems that OVN is not happy with the status of certificates. >>>> When testing connection at engine GUI i get a prompt to trust the >>>> cert, and when pressing ok i get a green confirmation of successful >>>> connection. >>>> >>>> >>> This is only relevant for the connection from oVirt Engine to >>> ovirt-provider-ovn. The prompt to trust the certificate might be redundant. >>> If you get the green confirmation, oVirt Engine is happy and the >>> certificate of the REST API of ovirt-provider-ovn is fine. >>> >>> >>>> Is there anything else that can be done to fix OVN functionality? >>>> >>> >>> Please try to understand what is wrong in the connection between >>> ovn-controller and ovn south bound db. >>> /var/log/openvswitch/ovn-controller.log should be helpful and might >>> contain the reason. >>> >> Will run the steps again to see. Do you think I need to take additional >> steps when fixing the OVN certs issue due to domain change that this >> cluster has undergone? >> > This time was not able to make OVN provider succeed at the test > connection with the new certs. Restored the pki to its previous state, > using the previous CA. Now, I do not see any errors for the last hour. > Created also from GUI a logical switch named ovn-switch-1. Attached two VMs > to it and was not able to confirm ping between VMs. > > At engine I see both the MAC addresses of each guest VM. > [root@engine ~]# ovn-nbctl show > switch ae4e03eb-e097-4629-a7bd-3272eee65599 > (ovirt-ovn-switch-1-bd04ad69-11a9-46d0-b571-f7dee62dfb7c) > port c4e4098b-764f-4696-8506-ccf46a535fd2 > addresses: ["00:1a:4a:16:02:59"] > port 14b8fb48-5ab1-4b10-93e1-d4e1fab17b51 > addresses: ["00:1a:4a:16:02:5e"] > > and the two hosts: > [root@engine ~]# ovn-sbctl show > Chassis "580a335e-f55f-4947-95d5-e90690b05125" > hostname: "v1" > Encap geneve > ip: "10.10.10.12" > options: {csum="true"} > Port_Binding "14b8fb48-5ab1-4b10-93e1-d4e1fab17b51" > Port_Binding "c4e4098b-764f-4696-8506-ccf46a535fd2" > Chassis "872e9ea7-fe6e-455f-8645-a2d6159c7552" > hostname: "v0" > Encap geneve > ip: "10.10.10.11" > options: {csum="true"} > > The status of OVN provider seem fine at engine: > [root@engine ~]# systemctl status ovirt-provider-ovn.service > ● ovirt-provider-ovn.service - oVirt OVN provider > Loaded: loaded (/usr/lib/systemd/system/ovirt-provider-ovn.service; > enabled; vendor preset: disabled) > Active: active (running) since Mon 2020-11-23 16:40:33 EET; 10min ago > Main PID: 25293 (python2) > Tasks: 4 > CGroup: /system.slice/ovirt-provider-ovn.service > └─25293 /usr/bin/python2 > /usr/share/ovirt-provider-ovn/ovirt_provider_ovn.py > > Nov 23 16:40:33 engine.mtis.tech systemd[1]: Started oVirt OVN provider. > Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - > [23/Nov/2020 16:44:26] "POST /v2.0//tokens HTTP/1.1" 200 - > Nov 23 16:44:26 engine.mtis.tech python2[25293]: ::ffff:10.10.10.13 - - > [23/Nov/2020 16:44:26] "GET /v2.0/networks HTTP/1.1" 200 - > > Also the status of OVN controller at each host seem fine: > [root@v0 ~]# systemctl status ovn-controller.service > ● ovn-controller.service - OVN controller daemon > Loaded: loaded (/usr/lib/systemd/system/ovn-controller.service; > enabled; vendor preset: disabled) > Active: active (running) since Sun 2020-11-22 11:43:17 EET; 1 day 5h > ago > Process: 11492 ExecStop=/usr/share/openvswitch/scripts/ovn-ctl > stop_controller (code=exited, status=0/SUCCESS) > Process: 11509 ExecStart=/usr/share/openvswitch/scripts/ovn-ctl > --no-monitor start_controller $OVN_CONTROLLER_OPTS (code=exited, > status=0/SUCCESS) > Main PID: 11539 (ovn-controller) > Tasks: 4 > CGroup: /system.slice/ovn-controller.service > └─11539 ovn-controller unix:/var/run/openvswitch/db.sock > -vconsole:emer -vsyslog:err -vfile:info --private-key=/etc/pki... > > Nov 22 11:43:17 v0.mtis.tech systemd[1]: Starting OVN controller daemon... > Nov 22 11:43:17 v0.mtis.tech ovn-ctl[11509]: Starting ovn-controller [ > OK ] > Nov 22 11:43:17 v0.mtis.tech systemd[1]: Started OVN controller daemon. > > What could be the reason of traffic not being forwarded through the > logical switch? > I noticed that there is a network port security setting when defining the network at the external provider, which is enabled by default. [image: image.png] Setting this to disabled made the VMs attached to the same network to be able to communicate through it.

It seems that when enabled this does drop all traffic by default if not specifically configured. I am wondering where I could configure a security group/policy to allow such traffic.

> > >>> >>> >>>> Thanx >>>> Alex >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Nov 19, 2020 at 9:00 AM Alex K <rightkicktech(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Seems that all services (imageio, ovn, web socket) are fine after >>>>> following the above and importing the new self signed CA certificate. >>>>> DId run also engine-setup as I was trying to fix the imageio cert >>>>> issue, though seems that that was only fixed after importing the CA cert at >>>>> browser and engine-setup might not be needed. >>>>> >>>>> On Wed, Nov 18, 2020 at 3:07 PM Alex K <rightkicktech(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Seems I had a typo at >>>>>> /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf. >>>>>> I will repeat the test to verify that all services are functional >>>>>> following this process. >>>>>> >>>>>> On Wed, Nov 18, 2020 at 10:24 AM Alex K <rightkicktech(a)gmail.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I am trying to replace the ovirt certificate at ovirt 4.3 following >>>>>>> this: >>>>>>> >>>>>>> >>>>>>> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/... >>>>>>> >>>>>>> I am doing the following: >>>>>>> I have engine FQDN: manager.lab.local >>>>>>> >>>>>>> 1. Create root CA private key: >>>>>>> openssl genrsa -des3 -out root.key 2048 >>>>>>> >>>>>>> 2. Generate root certificate: (enter passphrase of root key) >>>>>>> openssl req -x509 -new -nodes -key root.key -sha256 -days 3650 -out >>>>>>> root.pem >>>>>>> cp root.pem /tmp >>>>>>> >>>>>>> 3. Create key and CSR for engine: >>>>>>> openssl genrsa -out manager.lab.local.key 2048 >>>>>>> openssl req -new -out manager.lab.local.csr -key >>>>>>> manager.lab.local.key >>>>>>> >>>>>>> 4. Generate a certificate for engine and sign with the root CA key: >>>>>>> >>>>>>> openssl x509 -req -in manager.lab.local.csr \ >>>>>>> -CA root.pem \ >>>>>>> -CAkey root.key \ >>>>>>> -CAcreateserial \ >>>>>>> -out manager.lab.local.crt \ >>>>>>> -days 3650 \ >>>>>>> -sha256 \ >>>>>>> -extensions v3_req >>>>>>> >>>>>>> 5. Verify the trust chain and check the certificate details: >>>>>>> openssl verify -CAfile root.pem manager.lab.local.crt >>>>>>> openssl x509 -text -noout -in manager.lab.local.crt | head -15 >>>>>>> >>>>>>> 6. Generate a P12 container: (with empty password) >>>>>>> openssl pkcs12 -export -out /tmp/apache.p12 \ >>>>>>> -inkey manager.lab.local.key \ >>>>>>> -in manager.lab.local.crt >>>>>>> >>>>>>> 8. Export key and cert: >>>>>>> openssl pkcs12 -in apache.p12 -nocerts -nodes > /tmp/apache.key >>>>>>> openssl pkcs12 -in apache.p12 -nokeys > /tmp/apache.cer >>>>>>> >>>>>>> From the above steps we should have the following: >>>>>>> >>>>>>> /tmp/root.pem >>>>>>> /tmp/apache.p12 >>>>>>> /tmp/apache.key >>>>>>> /tmp/apache.cer >>>>>>> >>>>>>> 9. Place the certificates: >>>>>>> hosted-engine --set-maintenance --mode=global >>>>>>> cp -p /etc/pki/ovirt-engine/keys/apache.p12 /tmp/apache.p12.bck >>>>>>> cp /tmp/apache.p12 /etc/pki/ovirt-engine/keys/apache.p12 >>>>>>> cp /tmp/root.pem /etc/pki/ca-trust/source/anchors >>>>>>> update-ca-trust >>>>>>> rm /etc/pki/ovirt-engine/apache-ca.pem >>>>>>> cp /tmp/root.pem /etc/pki/ovirt-engine/apache-ca.pem >>>>>>> >>>>>>> Backup existing key and cert: >>>>>>> cp /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>>>> /etc/pki/ovirt-engine/keys/apache.key.nopass.bck >>>>>>> cp /etc/pki/ovirt-engine/certs/apache.cer >>>>>>> /etc/pki/ovirt-engine/certs/apache.cer.bck >>>>>>> cp /tmp/apache.key /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>>>> cp /tmp/apache.cer /etc/pki/ovirt-engine/certs/apache.cer >>>>>>> chown root:ovirt /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>>>> chmod 640 /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>>>> systemctl restart httpd.service >>>>>>> >>>>>>> 10. Create a new trust store configuration file: >>>>>>> vi /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf >>>>>>> >>>>>>> ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" >>>>>>> ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" >>>>>>> >>>>>>> 11. Edit >>>>>>> /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf : >>>>>>> vi /etc/ovirt-engine/ovirt-websocket-proxy.conf.d/10-setup.conf >>>>>>> >>>>>>> SSL_CERTIFICATE=/etc/pki/ovirt-engine/certs/apache.cer >>>>>>> SSL_KEY=/etc/pki/ovirt-engine/keys/apache.key.nopass >>>>>>> >>>>>>> 12. Edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf: >>>>>>> vi /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf >>>>>>> >>>>>>> # Key file for SSL connections >>>>>>> ssl_key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass >>>>>>> # Certificate file for SSL connections >>>>>>> ssl_cert_file = /etc/pki/ovirt-engine/certs/apache.cer >>>>>>> >>>>>>> 13. Import the certificate at system-wide java trust store >>>>>>> >>>>>>> update-ca-trust extract >>>>>>> keytool -list -alias ovirt -keystore /etc/pki/java/cacerts >>>>>>> >>>>>>> 14. Restart services: >>>>>>> systemctl restart httpd.service >>>>>>> systemctl restart ovirt-provider-ovn.service >>>>>>> systemctl restart ovirt-imageio-proxy >>>>>>> systemctl restart ovirt-websocket-proxy >>>>>>> systemctl restart ovirt-engine.service >>>>>>> >>>>>>> Following the above I get at engine GUI: >>>>>>> >>>>>>> sun.security.validator.ValidatorException: PKIX path building >>>>>>> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable >>>>>>> to find valid certification path to requested target >>>>>>> >>>>>>> I have tried also to run engine-setup in case it could fix anything >>>>>>> (it renewed the cert due to missing subjectAltName), and the above error >>>>>>> still persists. >>>>>>> I have tried several other suggestions from similar issues reported >>>>>>> at this list without any luck. >>>>>>> I have run out of ideas. Am I missing anything? >>>>>>> Thanx for any suggestions. >>>>>>> Alex >>>>>>> >>>>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/PKKBI7Y2RZB... >>>> >>>