[ovirt-devel] Re: test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)

Given the code freeze this week, could you please merge ASAP, so that we can run OST with other patches? Thanks Vojta On Monday, 22 February 2021 17:07:49 CET Artur Socha wrote: > And the fix for the engine is here: > https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/ > > Artur > > On 22.02.2021 16:29, Marcin Sobczyk wrote: > > Hi, > > > > On 2/22/21 4:21 PM, Yedidyah Bar David wrote: > >> On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha(a)redhat.com&gt; wrote: > >>> Hi Didi, > >>> You are probably right that enabling Strict Transport Security caused > >>> that bug as an unfortunate side-effect. > >>> Do you think that, adding some sort of exception for cert url would be > >>> an acceptable fix? For example we have this kind of rule for excluding > >>> authentication for Rest api docs. > >> > >> If we already have an exception, and hopefully some process to add one, > >> then I think it makes sense for this case as well. > >> > >> I admit, though, that I do not feel completely happy with this. On one > >> hand, > >> this is insecure, and on the other hand, there is no way to do this > >> securely > >> using the existing official means. > >> > >> This thread also made me think about the hosted-engine deploy process. > >> In standalone engine setup, the user is responsible for installing the > >> OS, > >> so it's up to the user to control (or not) generation of the sshd > >> private key > >> for allowing later secure access to it using ssh. For hosted-engine, > >> it's us, > >> and I do not think we do anything around this. Perhaps we should. > >> > >> TL;DR: IMO: > >> 1. Please add an exception. Please open another bug for this. > >> 2. We should document how to get the engine CA cert not using https: > >> ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . > >> 3. We should consider our options for hosted-engine. Filed now [1]. > >> > >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510 > >> > >> Best regards, > > > > For now I posted a patch for OST that will unblock basic suite [2]. > > When we have a proper solution we should adapt the tests to the new way > > of working. > > > > Regards, Marcin > > > > [2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/ > > > >>> Artur > >>> > >>> On 22.02.2021 13:52, Yedidyah Bar David wrote: > >>>> On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > >>>>> Project: > >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni > >>>>> ghtly/ > >>>>> > >>>>> Build: > >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni > >>>>> ghtly/894/ > >>>>> > >>>>> Build Number: 894 > >>>>> Build Status: Failure > >>>>> Triggered By: Started by timer > >>>>> > >>>>> ------------------------------------- > >>>>> Changes Since Last Success: > >>>>> ------------------------------------- > >>>>> Changes for Build #894 > >>>>> [Andrej Cernek] ost_utils: Remove explicit object inheritance > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> ----------------- > >>>>> Failed Tests: > >>>>> ----------------- > >>>>> 1 tests failed. > >>>>> FAILED: > >>>>> basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engin > >>>>> e_certs[CA certificate] > >>>>> > >>>>> Error Message: > >>>>> ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > >>>>> Stderr: unable to load certificate > >>>>> 139734854465344:error:0909006C:PEM routines:get_name:no start > >>>>> line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > >>>>> > >>>>> Stack Trace: > >>>>> key_format = 'X509-PEM-CA' > >>>>> verification_fn = <function <lambda> at 0x7f6aab2add90>, > >>>>> engine_fqdn = 'engine' > >>>>> engine_download = <function engine_download.<locals>.download at > >>>>> 0x7f6aa98d5ea0> > >>>>> > >>>>> @pytest.mark.parametrize("key_format, verification_fn", [ > >>>>> pytest.param( > >>>>> 'X509-PEM-CA', > >>>>> lambda path: shell.shell(["openssl", "x509", "-in", > >>>>> path, "-text", "-noout"]), > >>>>> id="CA certificate" > >>>>> ), > >>>>> pytest.param( > >>>>> 'OPENSSH-PUBKEY', > >>>>> lambda path: shell.shell(["ssh-keygen", "-l", "-f", > >>>>> path]), > >>>>> id="ssh pubkey" > >>>>> ), > >>>>> ]) > >>>>> @order_by(_TEST_LIST) > >>>>> def test_verify_engine_certs(key_format, verification_fn, > >>>>> engine_fqdn, > >>>>> engine_download): > >>>>> url = > >>>>> 'http:// {}/ovirt-engine/services/pki-resource?resource=ca-certificate& > >>>>> format={}'>>>> > >>>> I guess (didn't check, only looked at engine git log) that this is a > >>>> result of [1]. > >>>> > >>>> Anyone looking at this? > >>>> > >>>> This is trying to download the engine ca cert via http, and then do > >>>> some verification on it. > >>>> > >>>> Generally speaking, this is a chicken-and-egg problem: You can't > >>>> securely download > >>>> a ca cert if you need this cert to securely download it. > >>>> > >>>> For OST, it might be easy to fix by s/http/https/ and perhaps passing > >>>> some param to > >>>> make it not check certs in https. But I find it quite reasonable that > >>>> others are doing > >>>> similar things and will now be broken by this change [1]. If so, we > >>>> might decide that > >>>> this is "by design" - that whoever that gets broken, should fix their > >>>> stuff one way or > >>>> another (like OST above, or via safer means if possible/relevant, such > >>>> as using ssh > >>>> to securely connect to the engine machine and then get the cert from > >>>> there somehow > >>>> (do we have an api for this?)). Or we can decide that it's an engine > >>>> bug - that [1] > >>>> should have allowed this specific url to bypass hsts. > >>>> > >>>> [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 > >>>> > >>>>> with http_proxy_disabled(), tempfile.NamedTemporaryFile() > >>>>> as tmp: > >>>>> engine_download(url.format(engine_fqdn, key_format), > >>>>> tmp.name) > >>>>> > >>>>> try: > >>>>>> verification_fn(tmp.name) > >>>>> > >>>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >>>>> _ _ _ _ _ _ > >>>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in > >>>>> <lambda> > >>>>> lambda path: shell.shell(["openssl", "x509", "-in", path, > >>>>> "-text", "-noout"]), > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >>>>> _ _ _ _ _ _ > >>>>> > >>>>> args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', > >>>>> '-noout'] > >>>>> bytes_output = False, kwargs = {} > >>>>> process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > >>>>> err = 'unable to load > >>>>> certificate\n139734854465344:error:0909006C:PEM > >>>>> routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: > >>>>> TRUSTED CERTIFICATE\n' > >>>>> > >>>>> def shell(args, bytes_output=False, **kwargs): > >>>>> process = subprocess.Popen(args, > >>>>> stdout=subprocess.PIPE, > >>>>> stderr=subprocess.PIPE, > >>>>> **kwargs) > >>>>> out, err = process.communicate() > >>>>> > >>>>> if not bytes_output: > >>>>> out = out.decode("utf-8") > >>>>> err = err.decode("utf-8") > >>>>> > >>>>> if process.returncode: > >>>>>> raise ShellError(process.returncode, out, err) > >>>>> > >>>>> E ost_utils.shell.ShellError: Command failed with rc=1. > >>>>> Stdout: > >>>>> E > >>>>> E Stderr: > >>>>> E unable to load certificate > >>>>> E 139734854465344:error:0909006C:PEM routines:get_name:no > >>>>> start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > >>>> > >>>> (As I said, didn't check myself - I suppose that hsts causes httpd to > >>>> return some kind of redirect, and this is the way openssl fails when > >>>> we input this redirect instead of a cert). > >>>> > >>>> Best regards, _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/N72N67VDSY2...