Use public-signed SSL certs?
I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm... That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work. I'm trying to avoid ever needing to install a special CA cert in browsers. -- Chris Adams <cma@cmadams.net>
Did you try running engine-setup ? Regards, Paul S. ________________________________________ From: Chris Adams <cma@cmadams.net> Sent: 29 January 2019 15:51 To: users@ovirt.org Subject: [ovirt-users] Use public-signed SSL certs? I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm... That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work. I'm trying to avoid ever needing to install a special CA cert in browsers. -- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NZBGRCKW6WA4WI... To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/
I had not. Can that be run non-interactively to do whatever is needed? I'm using a Let's Encrypt cert, which needs to have a 100% automated deployment. Once upon a time, Staniforth, Paul <P.Staniforth@leedsbeckett.ac.uk> said:
Did you try running engine-setup ?
Regards, Paul S. ________________________________________ From: Chris Adams <cma@cmadams.net> Sent: 29 January 2019 15:51 To: users@ovirt.org Subject: [ovirt-users] Use public-signed SSL certs?
I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work.
I'm trying to avoid ever needing to install a special CA cert in browsers. -- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NZBGRCKW6WA4WI... To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/
-- Chris Adams <cma@cmadams.net>
On 1/29/19 1:30 PM, Chris Adams wrote:
Can that be run non-interactively to do whatever is needed? I'm using a Let's Encrypt cert, which needs to have a 100% automated deployment.
Yes, I believe so. Look at the whole biz with the "answers" file and the --config-append=file option. You should already have a generated answers file laying around from when you ran engine-setup before. See /var/lib/ovirt-engine/setup/answers IIRC.
Once upon a time, John Florian <jflorian@doubledog.org> said:
On 1/29/19 1:30 PM, Chris Adams wrote:
Can that be run non-interactively to do whatever is needed? I'm using a Let's Encrypt cert, which needs to have a 100% automated deployment.
Yes, I believe so. Look at the whole biz with the "answers" file and the --config-append=file option. You should already have a generated answers file laying around from when you ran engine-setup before. See /var/lib/ovirt-engine/setup/answers IIRC.
Hmm, that won't work - it looks like you can't run engine-setup on a hosted engine unless you first set hosted-engine HA to global maintenance. Is running engine-setup necessary to install/update certificates, or maybe is there a simpler way? -- Chris Adams <cma@cmadams.net>
On 1/29/19 2:47 PM, Chris Adams wrote:
On 1/29/19 1:30 PM, Chris Adams wrote:
Can that be run non-interactively to do whatever is needed? I'm using a Let's Encrypt cert, which needs to have a 100% automated deployment. Yes, I believe so. Look at the whole biz with the "answers" file and the --config-append=file option. You should already have a generated answers file laying around from when you ran engine-setup before. See /var/lib/ovirt-engine/setup/answers IIRC. Hmm, that won't work - it looks like you can't run engine-setup on a hosted engine unless you first set hosted-engine HA to global
Once upon a time, John Florian <jflorian@doubledog.org> said: maintenance.
Is running engine-setup necessary to install/update certificates, or maybe is there a simpler way?
I'm quite certain you can do it w/o engine-setup if you hit all the right file locations.
On 1/29/19 3:13 PM, John Florian wrote:
On 1/29/19 2:47 PM, Chris Adams wrote:
On 1/29/19 1:30 PM, Chris Adams wrote:
Can that be run non-interactively to do whatever is needed? I'm using a Let's Encrypt cert, which needs to have a 100% automated deployment. Yes, I believe so. Look at the whole biz with the "answers" file and the --config-append=file option. You should already have a generated answers file laying around from when you ran engine-setup before. See /var/lib/ovirt-engine/setup/answers IIRC. Hmm, that won't work - it looks like you can't run engine-setup on a hosted engine unless you first set hosted-engine HA to global
Once upon a time, John Florian <jflorian@doubledog.org> said: maintenance.
Is running engine-setup necessary to install/update certificates, or maybe is there a simpler way?
I'm quite certain you can do it w/o engine-setup if you hit all the right file locations.
Just to follow up on this Chris, I have my puppet drop my CA cert in /etc/pki/ca-trust/source/anchors/, my self-signed cert in/etc/pki/ovirt-engine/certs/ and my key in /etc/pki/ovirt-engine/keys. I also manage /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf to have: ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD="" I believe this gives me everything you seek. -- John Florian
Once upon a time, John Florian <jflorian@doubledog.org> said:
Just to follow up on this Chris, I have my puppet drop my CA cert in /etc/pki/ca-trust/source/anchors/, my self-signed cert in/etc/pki/ovirt-engine/certs/ and my key in /etc/pki/ovirt-engine/keys. I also manage /etc/ovirt-engine/engine.conf.d/99-custom-truststore.conf to have:
ENGINE_HTTPS_PKI_TRUST_STORE="/etc/pki/java/cacerts" ENGINE_HTTPS_PKI_TRUST_STORE_PASSWORD=""
I believe this gives me everything you seek.
That works to get the core engine UI using a new cert (that and a little more are in the Red Hat URL in my original message). It doesn't handle the imageio-proxy however. -- Chris Adams <cma@cmadams.net>
On Tue, Jan 29, 2019 at 6:05 PM Chris Adams <cma@cmadams.net> wrote:
I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work.
Did you restart the imageio-proxy? What didn't work? What happened?
I'm trying to avoid ever needing to install a special CA cert in browsers.
Makes sense. This is known bug: https://bugzilla.redhat.com/show_bug.cgi?id=1637809 Before opening it, we had a bug about fixing the documentation you point at: https://bugzilla.redhat.com/show_bug.cgi?id=1385617 As mentioned there, what you tried to do should have worked. Best regards, -- Didi
Once upon a time, Yedidyah Bar David <didi@redhat.com> said:
On Tue, Jan 29, 2019 at 6:05 PM Chris Adams <cma@cmadams.net> wrote:
I installed an SSL cert from a public CA (Let's Encrypt) on my engine, following this:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
That gets the regular web UI working, but I can't upload an ISO. I assume that I need to do something with the imageio-proxy service on the engine, but not sure what... I tried replacing imageio-proxy.cer and imageio-proxy.key.nopass, but that didn't work.
Did you restart the imageio-proxy?
What didn't work? What happened?
I did restart the service. When I then try to upload an ISO image, I get "Paused by System" and this in engine.log: ######################################################################## 2019-01-30 08:12:15,871-06 ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [0052c7ad-38d7-429d-be3a-eb0e496d5ee8] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_191] at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_191] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_191] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_191] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_191] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_191] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_191] at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_191] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_191] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_191] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_191] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_191] at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_191] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_191] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_191] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_191] at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_191] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommand.addImageTicketToProxy(TransferImageCommand.java:654) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommand.startImageTransferSession(TransferImageCommand.java:579) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommand.handleImageIsReadyForTransfer(TransferImageCommand.java:261) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommand.handleInitializing(TransferImageCommand.java:232) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommand.executeStateHandler(TransferImageCommand.java:167) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommand.proceedCommandExecution(TransferImageCommand.java:154) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.TransferImageCommandCallback.doPolling(TransferImageCommandCallback.java:21) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:146) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:107) [bll.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_191] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [rt.jar:1.8.0_191] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:383) [javax.enterprise.concurrent-1.0.jar:] at org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:534) [javax.enterprise.concurrent-1.0.jar:] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191] at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent-1.0.jar:] at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_191] at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_191] at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_191] at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_191] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_191] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_191] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_191] ... 30 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_191] at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_191] at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_191] at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_191] ... 36 more ######################################################################## I'm guessing that I affected the engine's ability to validate the public-CA-signed cert on the imageio-proxy? Maybe I just messed something else up?
I'm trying to avoid ever needing to install a special CA cert in browsers.
Makes sense.
This is known bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1637809
Before opening it, we had a bug about fixing the documentation you point at:
https://bugzilla.redhat.com/show_bug.cgi?id=1385617
As mentioned there, what you tried to do should have worked.
I saw the second BZ and read through it. I was taking the approach of replacing the imageio-proxy key/cert rather than repointing it; I've switched to just changing the config but have the same issue. -- Chris Adams <cma@cmadams.net>
Digging a little deeper... if I add the Let's Encrypt CA to /etc/pki/ovirt-engine/.truststore, imageio-proxy works (I can successfully upload an ISO), so I guess the issue is that imageio-proxy uses the same cert for web and engine communication and the engine wasn't happy with the public-CA-signed cert. So, rather than point part of the engine at a separate trust store (as the docs recommend), maybe just add the public CA to the engine's existing trust store? However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert. This is all on 4.2.8 BTW. -- Chris Adams <cma@cmadams.net>
I got it working by changing the SSL certs pointed out in the /etc/imageio-proxy/imageio-proxy.conf. BR Chris Adams <cma@cmadams.net> schrieb am Mi., 30. Jan. 2019, 21:28:
Digging a little deeper... if I add the Let's Encrypt CA to /etc/pki/ovirt-engine/.truststore, imageio-proxy works (I can successfully upload an ISO), so I guess the issue is that imageio-proxy uses the same cert for web and engine communication and the engine wasn't happy with the public-CA-signed cert.
So, rather than point part of the engine at a separate trust store (as the docs recommend), maybe just add the public CA to the engine's existing trust store?
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
This is all on 4.2.8 BTW. -- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FC6FNKINSVQFA7...
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
Digging a little deeper... if I add the Let's Encrypt CA to /etc/pki/ovirt-engine/.truststore, imageio-proxy works (I can successfully upload an ISO), so I guess the issue is that imageio-proxy uses the same cert for web and engine communication and the engine wasn't happy with the public-CA-signed cert.
I think I agree with your analysis. I now reproduced this on a test env. I started with ovirt-system-tests basic suite deploy, made sure I can upload an image. Then I followed the docs about replacing certs, using a temporarily- created CA for testing (using openssl, actually using a copy of the engine's pki scripts), including adding 99-custom-truststore.conf, imported the CA's cert to the browser, and: 1. Connecting with the browser worked, all is green. 2. Logged in, pressed "Disks -> Upload -> Start -> Test Connection", and it failed. 3. Edited the ovirt-imageio-proxy conf to point key and cert to a key and cert I created and signed using my temp ca, restarted it, "Test Connection" worked. 4. Actually uploading the image failed as you describe. 5. Imported my CA's cert to /etc/pki/ovirt-engine/.truststore, using: keytool -importcert -trustcacerts -keystore /etc/pki/ovirt-engine/.truststore -storepass mypass -file /etc/pki/ovirt-engine/apache-ca.pem and restarted the engine, and then upload works. Adding Martin and Nir.
So, rather than point part of the engine at a separate trust store (as the docs recommend), maybe just add the public CA to the engine's existing trust store?
I admit I still didn't try to fully analyze this myself, but I tend to agree with you. Or rather: Our docs should probably support both options - tell the engine to trust (and use?) the system-wide store, or manually add a specific cert. Because I guess you can find people that will prefer either option.
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
This is all on 4.2.8 BTW.
I personally tried this on: ovirt-engine-4.3.0-0.8.master.20190122121624.git9a8a519.el7.noarch I guess the behavior didn't change much between them. Thanks for your debugging and report! Best regards, -- Didi
On Mon, 4 Feb 2019 13:21:56 +0200 Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
Digging a little deeper... if I add the Let's Encrypt CA to /etc/pki/ovirt-engine/.truststore, imageio-proxy works (I can successfully upload an ISO), so I guess the issue is that imageio-proxy uses the same cert for web and engine communication and the engine wasn't happy with the public-CA-signed cert.
I think I agree with your analysis.
I now reproduced this on a test env.
I started with ovirt-system-tests basic suite deploy, made sure I can upload an image.
Then I followed the docs about replacing certs, using a temporarily- created CA for testing (using openssl, actually using a copy of the engine's pki scripts), including adding 99-custom-truststore.conf, imported the CA's cert to the browser, and:
1. Connecting with the browser worked, all is green.
2. Logged in, pressed "Disks -> Upload -> Start -> Test Connection", and it failed.
3. Edited the ovirt-imageio-proxy conf to point key and cert to a key and cert I created and signed using my temp ca, restarted it, "Test Connection" worked.
4. Actually uploading the image failed as you describe.
5. Imported my CA's cert to /etc/pki/ovirt-engine/.truststore, using:
keytool -importcert -trustcacerts -keystore /etc/pki/ovirt-engine/.truststore -storepass mypass -file /etc/pki/ovirt-engine/apache-ca.pem
and restarted the engine, and then upload works.
Adding Martin and Nir.
So, rather than point part of the engine at a separate trust store (as the docs recommend), maybe just add the public CA to the engine's existing trust store?
I admit I still didn't try to fully analyze this myself, but I tend to agree with you. Or rather: Our docs should probably support both options - tell the engine to trust (and use?) the system-wide store, or manually add a specific cert. Because I guess you can find people that will prefer either option.
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
Please ensure that the configured certificates in /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf especially ovirt-ca-file, points to the expected files and restart ovirt-provider-ovn. If this does not solve the issue, please share ovirt-provider-ovn.log.
This is all on 4.2.8 BTW.
I personally tried this on:
ovirt-engine-4.3.0-0.8.master.20190122121624.git9a8a519.el7.noarch
I guess the behavior didn't change much between them.
Thanks for your debugging and report!
Best regards,
On Mon, Feb 4, 2019 at 1:21 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
Digging a little deeper... if I add the Let's Encrypt CA to /etc/pki/ovirt-engine/.truststore, imageio-proxy works (I can successfully upload an ISO), so I guess the issue is that imageio-proxy uses the same cert for web and engine communication and the engine wasn't happy with the public-CA-signed cert.
I think I agree with your analysis.
I now reproduced this on a test env.
I started with ovirt-system-tests basic suite deploy, made sure I can upload an image.
Then I followed the docs about replacing certs, using a temporarily- created CA for testing (using openssl, actually using a copy of the engine's pki scripts), including adding 99-custom-truststore.conf, imported the CA's cert to the browser, and:
1. Connecting with the browser worked, all is green.
2. Logged in, pressed "Disks -> Upload -> Start -> Test Connection", and it failed.
3. Edited the ovirt-imageio-proxy conf to point key and cert to a key and cert I created and signed using my temp ca, restarted it, "Test Connection" worked.
4. Actually uploading the image failed as you describe.
5. Imported my CA's cert to /etc/pki/ovirt-engine/.truststore, using:
keytool -importcert -trustcacerts -keystore /etc/pki/ovirt-engine/.truststore -storepass mypass -file /etc/pki/ovirt-engine/apache-ca.pem
and restarted the engine, and then upload works.
Adding Martin and Nir.
So, rather than point part of the engine at a separate trust store (as the docs recommend), maybe just add the public CA to the engine's existing trust store?
I admit I still didn't try to fully analyze this myself, but I tend to agree with you. Or rather: Our docs should probably support both options - tell the engine to trust (and use?) the system-wide store, or manually add a specific cert. Because I guess you can find people that will prefer either option.
Decided that only the first makes sense, opened this bug, should be fixed in 4.3.2: https://bugzilla.redhat.com/1687301 This is obviously just one step. The next will be: https://bugzilla.redhat.com/show_bug.cgi?id=1637809 Then, hopefully, following the existing doc to use 3rd-party CA will "just work" also for imageio. BTW, of course you can also create another custom truststore only for https access to the engine, and point ENGINE_HTTPS_PKI_TRUST_STORE at it - but I wouldn't add this to the docs before we have automated testing that makes sure this does not break in the future. Best regards,
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
This is all on 4.2.8 BTW.
I personally tried this on:
ovirt-engine-4.3.0-0.8.master.20190122121624.git9a8a519.el7.noarch
I guess the behavior didn't change much between them.
Thanks for your debugging and report!
Best regards, -- Didi
-- Didi
Circling back to an old email... Once upon a time, Yedidyah Bar David <didi@redhat.com> said:
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
Was anybody able to look at this? I had to use my dev hardware for something else for a bit, so re-installed with 4.3.5 yesterday. The imageio SSL cert issue looks good, but I still can't figure out the ovirt-provider-ovn CA usage. My little bit of digging seems to show that the engine connects to the provider and is using an SSL client cert, and that cert is signed by something... but I'm not sure what. I think the provider side is trying to validate with the following setting from /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf [OVIRT] ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem Following the general "3rd-party SSL", that is now the Let's Encrypt CA. I tried changing it to point to the original self-signed oVirt CA (same directory, just "ca.pem"), but that didn't work either. Any suggestions? -- Chris Adams <cma@cmadams.net>
I figured it out. When ovirt-provider-ovn attempts to connect back to the engine via HTTPS, it tells the python requests module to use the specified CA cert file... but that won't work with most 3rd-party certs because they have an intermediate cert as well. It appears that the requests module tries to validate both certs. Creating /etc/ovirt-provider-ovn/conf.d/99-custom-cert.conf that just has: [OVIRT] ovirt-ca-file= tells the module to use the regular system CA cert file(s), which works. This should probably be added to the oVirt doc for using a 3rd-party cert. Once upon a time, Chris Adams <cma@cmadams.net> said:
Circling back to an old email...
Once upon a time, Yedidyah Bar David <didi@redhat.com> said:
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
Was anybody able to look at this? I had to use my dev hardware for something else for a bit, so re-installed with 4.3.5 yesterday. The imageio SSL cert issue looks good, but I still can't figure out the ovirt-provider-ovn CA usage.
My little bit of digging seems to show that the engine connects to the provider and is using an SSL client cert, and that cert is signed by something... but I'm not sure what. I think the provider side is trying to validate with the following setting from /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
[OVIRT] ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
Following the general "3rd-party SSL", that is now the Let's Encrypt CA. I tried changing it to point to the original self-signed oVirt CA (same directory, just "ca.pem"), but that didn't work either.
Any suggestions?
-- Chris Adams <cma@cmadams.net>
On Thu, 1 Aug 2019 20:45:56 -0500 Chris Adams <cma@cmadams.net> wrote:
I figured it out. When ovirt-provider-ovn attempts to connect back to the engine via HTTPS, it tells the python requests module to use the specified CA cert file... but that won't work with most 3rd-party certs because they have an intermediate cert as well. It appears that the requests module tries to validate both certs.
Creating /etc/ovirt-provider-ovn/conf.d/99-custom-cert.conf that just has:
[OVIRT] ovirt-ca-file=
tells the module to use the regular system CA cert file(s), which works.
Thanks for your investigation! Looks like the empty string is converted implicitly to Boolean in https://github.com/psf/requests/blob/75bdc998e2d430a35d869b2abf1779bd0d34890... Because bool('') is False in python, the certificate should be checked at all. Would ovirt-ca-file=/etc/pki/tls/certs/ca-bundle.crt work for you? (It works for https://helloworld.letsencrypt.org)
This should probably be added to the oVirt doc for using a 3rd-party cert.
Once upon a time, Chris Adams <cma@cmadams.net> said:
Circling back to an old email...
Once upon a time, Yedidyah Bar David <didi@redhat.com> said:
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
Was anybody able to look at this? I had to use my dev hardware for something else for a bit, so re-installed with 4.3.5 yesterday. The imageio SSL cert issue looks good, but I still can't figure out the ovirt-provider-ovn CA usage.
My little bit of digging seems to show that the engine connects to the provider and is using an SSL client cert, and that cert is signed by something... but I'm not sure what. I think the provider side is trying to validate with the following setting from /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
[OVIRT] ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
Following the general "3rd-party SSL", that is now the Let's Encrypt CA. I tried changing it to point to the original self-signed oVirt CA (same directory, just "ca.pem"), but that didn't work either.
Any suggestions?
On Fri, Aug 2, 2019 at 4:37 PM Dominik Holler <dholler@redhat.com> wrote:
On Thu, 1 Aug 2019 20:45:56 -0500 Chris Adams <cma@cmadams.net> wrote:
I figured it out. When ovirt-provider-ovn attempts to connect back to the engine via HTTPS, it tells the python requests module to use the specified CA cert file... but that won't work with most 3rd-party certs because they have an intermediate cert as well. It appears that the requests module tries to validate both certs.
Creating /etc/ovirt-provider-ovn/conf.d/99-custom-cert.conf that just has:
[OVIRT] ovirt-ca-file=
tells the module to use the regular system CA cert file(s), which works.
Thanks for your investigation! Looks like the empty string is converted implicitly to Boolean in
https://github.com/psf/requests/blob/75bdc998e2d430a35d869b2abf1779bd0d34890... Because bool('') is False in python, the certificate should be checked at all.
Because bool('') is False in python, the certificate should be* not *checked at all.
Would ovirt-ca-file=/etc/pki/tls/certs/ca-bundle.crt work for you? (It works for https://helloworld.letsencrypt.org)
This should probably be added to the oVirt doc for using a 3rd-party cert.
Circling back to an old email...
Once upon a time, Yedidyah Bar David <didi@redhat.com> said:
On Wed, Jan 30, 2019 at 10:28 PM Chris Adams <cma@cmadams.net> wrote:
However, while digging, I also noticed that now the engine is not communicating with ovirt-provider-ovn, possibly due to a similar issue? It is having the reverse problem; it rejects the engine's cert.
Didn't try this yet, adding Dominik.
Was anybody able to look at this? I had to use my dev hardware for something else for a bit, so re-installed with 4.3.5 yesterday. The imageio SSL cert issue looks good, but I still can't figure out the ovirt-provider-ovn CA usage.
My little bit of digging seems to show that the engine connects to the provider and is using an SSL client cert, and that cert is signed by something... but I'm not sure what. I think the provider side is
Once upon a time, Chris Adams <cma@cmadams.net> said: trying
to validate with the following setting from /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf
[OVIRT] ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem
Following the general "3rd-party SSL", that is now the Let's Encrypt CA. I tried changing it to point to the original self-signed oVirt CA (same directory, just "ca.pem"), but that didn't work either.
Any suggestions?
Once upon a time, Dominik Holler <dholler@redhat.com> said:
Would ovirt-ca-file=/etc/pki/tls/certs/ca-bundle.crt work for you?
Yes, that looks like it works correctly. Still chasing issues with a 3rd-party cert down... now it seems like there may be an SSL issue between ovirt-provider-ovn and ovsdb-server (seeing SSL and protocol errors in ovsdb-server-nb.log that weren't there before changing the cert). Also, I updated the engine from 4.3.4 to 4.3.5 and it overwrote the necessary changes in ovirt-imageio-proxy's config. -- Chris Adams <cma@cmadams.net>
participants (6)
-
Chris Adams -
Dominik Holler -
John Florian -
Sandro Emma -
Staniforth, Paul -
Yedidyah Bar David