[ovirt-devel] Re: test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)

On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha(a)redhat.com&gt; wrote: > Hi Didi, > You are probably right that enabling Strict Transport Security caused > that bug as an unfortunate side-effect. > Do you think that, adding some sort of exception for cert url would be > an acceptable fix? For example we have this kind of rule for excluding > authentication for Rest api docs. If we already have an exception, and hopefully some process to add one, then I think it makes sense for this case as well. I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means. This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should. TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510 Best regards,

> Artur > > > > > On 22.02.2021 13:52, Yedidyah Bar David wrote: >> On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >>> Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ >>> Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... >>> Build Number: 894 >>> Build Status: Failure >>> Triggered By: Started by timer >>> >>> ------------------------------------- >>> Changes Since Last Success: >>> ------------------------------------- >>> Changes for Build #894 >>> [Andrej Cernek] ost_utils: Remove explicit object inheritance >>> >>> >>> >>> >>> ----------------- >>> Failed Tests: >>> ----------------- >>> 1 tests failed. >>> FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate] >>> >>> Error Message: >>> ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE >>> >>> Stack Trace: >>> key_format = 'X509-PEM-CA' >>> verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' >>> engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0> >>> >>> @pytest.mark.parametrize("key_format, verification_fn", [ >>> pytest.param( >>> 'X509-PEM-CA', >>> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), >>> id="CA certificate" >>> ), >>> pytest.param( >>> 'OPENSSH-PUBKEY', >>> lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), >>> id="ssh pubkey" >>> ), >>> ]) >>> @order_by(_TEST_LIST) >>> def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, >>> engine_download): >>> url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}' >> I guess (didn't check, only looked at engine git log) that this is a >> result of [1]. >> >> Anyone looking at this? >> >> This is trying to download the engine ca cert via http, and then do >> some verification on it. >> >> Generally speaking, this is a chicken-and-egg problem: You can't >> securely download >> a ca cert if you need this cert to securely download it. >> >> For OST, it might be easy to fix by s/http/https/ and perhaps passing >> some param to >> make it not check certs in https. But I find it quite reasonable that >> others are doing >> similar things and will now be broken by this change [1]. If so, we >> might decide that >> this is "by design" - that whoever that gets broken, should fix their >> stuff one way or >> another (like OST above, or via safer means if possible/relevant, such >> as using ssh >> to securely connect to the engine machine and then get the cert from >> there somehow >> (do we have an api for this?)). Or we can decide that it's an engine >> bug - that [1] >> should have allowed this specific url to bypass hsts. >> >> [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 >> >>> with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: >>> engine_download(url.format(engine_fqdn, key_format), tmp.name) >>> try: >>>> verification_fn(tmp.name) >>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: >>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> >>> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), >>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>> >>> args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] >>> bytes_output = False, kwargs = {} >>> process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' >>> err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n' >>> >>> def shell(args, bytes_output=False, **kwargs): >>> process = subprocess.Popen(args, >>> stdout=subprocess.PIPE, >>> stderr=subprocess.PIPE, >>> **kwargs) >>> out, err = process.communicate() >>> >>> if not bytes_output: >>> out = out.decode("utf-8") >>> err = err.decode("utf-8") >>> >>> if process.returncode: >>>> raise ShellError(process.returncode, out, err) >>> E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: >>> E >>> E Stderr: >>> E unable to load certificate >>> E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE >> (As I said, didn't check myself - I suppose that hsts causes httpd to >> return some kind of redirect, and this is the way openssl fails when >> we input this redirect instead of a cert). >> >> Best regards, >>