[ovirt-devel] Re: test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)

And the fix for the engine is here: https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/ Artur On 22.02.2021 16:29, Marcin Sobczyk wrote: > Hi, > > On 2/22/21 4:21 PM, Yedidyah Bar David wrote: >> On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha(a)redhat.com&gt; wrote: >>> Hi Didi, >>> You are probably right that enabling Strict Transport Security caused >>> that bug as an unfortunate side-effect. >>> Do you think that, adding some sort of exception for cert url would be >>> an acceptable fix? For example we have this kind of rule for excluding >>> authentication for Rest api docs. >> >> If we already have an exception, and hopefully some process to add one, >> then I think it makes sense for this case as well. >> >> I admit, though, that I do not feel completely happy with this. On one >> hand, >> this is insecure, and on the other hand, there is no way to do this >> securely >> using the existing official means. >> >> This thread also made me think about the hosted-engine deploy process. >> In standalone engine setup, the user is responsible for installing the >> OS, >> so it's up to the user to control (or not) generation of the sshd >> private key >> for allowing later secure access to it using ssh. For hosted-engine, >> it's us, >> and I do not think we do anything around this. Perhaps we should. >> >> TL;DR: IMO: >> 1. Please add an exception. Please open another bug for this. >> 2. We should document how to get the engine CA cert not using https: >> ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . >> 3. We should consider our options for hosted-engine. Filed now [1]. >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510 >> >> Best regards, > > For now I posted a patch for OST that will unblock basic suite [2]. > When we have a proper solution we should adapt the tests to the new way > of working. > > Regards, Marcin > > [2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/ > >>> Artur >>> >>> On 22.02.2021 13:52, Yedidyah Bar David wrote: >>>> On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >>>>> Project: >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni >>>>> ghtly/ >>>>> >>>>> Build: >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni >>>>> ghtly/894/ >>>>> >>>>> Build Number: 894 >>>>> Build Status: Failure >>>>> Triggered By: Started by timer >>>>> >>>>> ------------------------------------- >>>>> Changes Since Last Success: >>>>> ------------------------------------- >>>>> Changes for Build #894 >>>>> [Andrej Cernek] ost_utils: Remove explicit object inheritance >>>>> >>>>> >>>>> >>>>> >>>>> ----------------- >>>>> Failed Tests: >>>>> ----------------- >>>>> 1 tests failed. >>>>> FAILED: >>>>> basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engin >>>>> e_certs[CA certificate] >>>>> >>>>> Error Message: >>>>> ost_utils.shell.ShellError: Command failed with rc=1. Stdout: >>>>> Stderr: unable to load certificate >>>>> 139734854465344:error:0909006C:PEM routines:get_name:no start >>>>> line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE >>>>> >>>>> Stack Trace: >>>>> key_format = 'X509-PEM-CA' >>>>> verification_fn = <function <lambda> at 0x7f6aab2add90>, >>>>> engine_fqdn = 'engine' >>>>> engine_download = <function engine_download.<locals>.download at >>>>> 0x7f6aa98d5ea0> >>>>> >>>>> @pytest.mark.parametrize("key_format, verification_fn", [ >>>>> pytest.param( >>>>> 'X509-PEM-CA', >>>>> lambda path: shell.shell(["openssl", "x509", "-in", >>>>> path, "-text", "-noout"]), >>>>> id="CA certificate" >>>>> ), >>>>> pytest.param( >>>>> 'OPENSSH-PUBKEY', >>>>> lambda path: shell.shell(["ssh-keygen", "-l", "-f", >>>>> path]), >>>>> id="ssh pubkey" >>>>> ), >>>>> ]) >>>>> @order_by(_TEST_LIST) >>>>> def test_verify_engine_certs(key_format, verification_fn, >>>>> engine_fqdn, >>>>> engine_download): >>>>> url = >>>>> 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate& >>>>> format={}'>>>> >>>> I guess (didn't check, only looked at engine git log) that this is a >>>> result of [1]. >>>> >>>> Anyone looking at this? >>>> >>>> This is trying to download the engine ca cert via http, and then do >>>> some verification on it. >>>> >>>> Generally speaking, this is a chicken-and-egg problem: You can't >>>> securely download >>>> a ca cert if you need this cert to securely download it. >>>> >>>> For OST, it might be easy to fix by s/http/https/ and perhaps passing >>>> some param to >>>> make it not check certs in https. But I find it quite reasonable that >>>> others are doing >>>> similar things and will now be broken by this change [1]. If so, we >>>> might decide that >>>> this is "by design" - that whoever that gets broken, should fix their >>>> stuff one way or >>>> another (like OST above, or via safer means if possible/relevant, such >>>> as using ssh >>>> to securely connect to the engine machine and then get the cert from >>>> there somehow >>>> (do we have an api for this?)). Or we can decide that it's an engine >>>> bug - that [1] >>>> should have allowed this specific url to bypass hsts. >>>> >>>> [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 >>>> >>>>> with http_proxy_disabled(), tempfile.NamedTemporaryFile() >>>>> as tmp: >>>>> engine_download(url.format(engine_fqdn, key_format), >>>>> tmp.name) >>>>> >>>>> try: >>>>>> verification_fn(tmp.name) >>>>> >>>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>>>> _ _ _ _ _ _ >>>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in >>>>> <lambda> >>>>> lambda path: shell.shell(["openssl", "x509", "-in", path, >>>>> "-text", "-noout"]), >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>>>> _ _ _ _ _ _ >>>>> >>>>> args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', >>>>> '-noout'] >>>>> bytes_output = False, kwargs = {} >>>>> process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' >>>>> err = 'unable to load >>>>> certificate\n139734854465344:error:0909006C:PEM >>>>> routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: >>>>> TRUSTED CERTIFICATE\n' >>>>> >>>>> def shell(args, bytes_output=False, **kwargs): >>>>> process = subprocess.Popen(args, >>>>> stdout=subprocess.PIPE, >>>>> stderr=subprocess.PIPE, >>>>> **kwargs) >>>>> out, err = process.communicate() >>>>> >>>>> if not bytes_output: >>>>> out = out.decode("utf-8") >>>>> err = err.decode("utf-8") >>>>> >>>>> if process.returncode: >>>>>> raise ShellError(process.returncode, out, err) >>>>> >>>>> E ost_utils.shell.ShellError: Command failed with rc=1. >>>>> Stdout: >>>>> E >>>>> E Stderr: >>>>> E unable to load certificate >>>>> E 139734854465344:error:0909006C:PEM routines:get_name:no >>>>> start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE >>>> >>>> (As I said, didn't check myself - I suppose that hsts causes httpd to >>>> return some kind of redirect, and this is the way openssl fails when >>>> we input this redirect instead of a cert). >>>> >>>> Best regards,