[ovirt-devel] Re: test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)

Hi, On 2/22/21 4:21 PM, Yedidyah Bar David wrote: > On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha(a)redhat.com&gt; wrote: >> Hi Didi, >> You are probably right that enabling Strict Transport Security caused >> that bug as an unfortunate side-effect. >> Do you think that, adding some sort of exception for cert url would be >> an acceptable fix?  For example we have this kind of rule for excluding >> authentication for Rest api docs. > If we already have an exception, and hopefully some process to add one, > then I think it makes sense for this case as well. > > I admit, though, that I do not feel completely happy with this. On one > hand, > this is insecure, and on the other hand, there is no way to do this > securely > using the existing official means. > > This thread also made me think about the hosted-engine deploy process. > In standalone engine setup, the user is responsible for installing the > OS, > so it's up to the user to control (or not) generation of the sshd > private key > for allowing later secure access to it using ssh. For hosted-engine, > it's us, > and I do not think we do anything around this. Perhaps we should. > > TL;DR: IMO: > 1. Please add an exception. Please open another bug for this. > 2. We should document how to get the engine CA cert not using https: > ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . > 3. We should consider our options for hosted-engine. Filed now [1]. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510 > > Best regards, For now I posted a patch for OST that will unblock basic suite [2]. When we have a proper solution we should adapt the tests to the new way of working. Regards, Marcin [2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/ > >> Artur >> >> >> >> >> On 22.02.2021 13:52, Yedidyah Bar David wrote: >>> On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >>>> Project: >>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ >>>> >>>> Build: >>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... >>>> >>>> Build Number: 894 >>>> Build Status:  Failure >>>> Triggered By: Started by timer >>>> >>>> ------------------------------------- >>>> Changes Since Last Success: >>>> ------------------------------------- >>>> Changes for Build #894 >>>> [Andrej Cernek] ost_utils: Remove explicit object inheritance >>>> >>>> >>>> >>>> >>>> ----------------- >>>> Failed Tests: >>>> ----------------- >>>> 1 tests failed. >>>> FAILED:  >>>> basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA >>>> certificate] >>>> >>>> Error Message: >>>> ost_utils.shell.ShellError: Command failed with rc=1. Stdout:  >>>> Stderr: unable to load certificate >>>> 139734854465344:error:0909006C:PEM routines:get_name:no start >>>> line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE >>>> >>>> Stack Trace: >>>> key_format = 'X509-PEM-CA' >>>> verification_fn = <function <lambda> at 0x7f6aab2add90>, >>>> engine_fqdn = 'engine' >>>> engine_download = <function engine_download.<locals>.download at >>>> 0x7f6aa98d5ea0> >>>> >>>>      @pytest.mark.parametrize("key_format, verification_fn", [ >>>>          pytest.param( >>>>              'X509-PEM-CA', >>>>              lambda path: shell.shell(["openssl", "x509", "-in", >>>> path, "-text", "-noout"]), >>>>              id="CA certificate" >>>>          ), >>>>          pytest.param( >>>>              'OPENSSH-PUBKEY', >>>>              lambda path: shell.shell(["ssh-keygen", "-l", "-f", >>>> path]), >>>>              id="ssh pubkey" >>>>          ), >>>>      ]) >>>>      @order_by(_TEST_LIST) >>>>      def test_verify_engine_certs(key_format, verification_fn, >>>> engine_fqdn, >>>>                                   engine_download): >>>>          url = >>>> 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}' >>>> >>> I guess (didn't check, only looked at engine git log) that this is a >>> result of [1]. >>> >>> Anyone looking at this? >>> >>> This is trying to download the engine ca cert via http, and then do >>> some verification on it. >>> >>> Generally speaking, this is a chicken-and-egg problem: You can't >>> securely download >>> a ca cert if you need this cert to securely download it. >>> >>> For OST, it might be easy to fix by s/http/https/ and perhaps passing >>> some param to >>> make it not check certs in https. But I find it quite reasonable that >>> others are doing >>> similar things and will now be broken by this change [1]. If so, we >>> might decide that >>> this is "by design" - that whoever that gets broken, should fix their >>> stuff one way or >>> another (like OST above, or via safer means if possible/relevant, such >>> as using ssh >>> to securely connect to the engine machine and then get the cert from >>> there somehow >>> (do we have an api for this?)). Or we can decide that it's an engine >>> bug - that [1] >>> should have allowed this specific url to bypass hsts. >>> >>> [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 >>> >>>>          with http_proxy_disabled(), tempfile.NamedTemporaryFile() >>>> as tmp: >>>>              engine_download(url.format(engine_fqdn, key_format), >>>> tmp.name) >>>>              try: >>>>>                verification_fn(tmp.name) >>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: >>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>>> _ _ _ _ _ _ >>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in >>>> <lambda> >>>>      lambda path: shell.shell(["openssl", "x509", "-in", path, >>>> "-text", "-noout"]), >>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>>> _ _ _ _ _ _ >>>> >>>> args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', >>>> '-noout'] >>>> bytes_output = False, kwargs = {} >>>> process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' >>>> err = 'unable to load >>>> certificate\n139734854465344:error:0909006C:PEM >>>> routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: >>>> TRUSTED CERTIFICATE\n' >>>> >>>>      def shell(args, bytes_output=False, **kwargs): >>>>          process = subprocess.Popen(args, >>>>                                     stdout=subprocess.PIPE, >>>>                                     stderr=subprocess.PIPE, >>>>                                     **kwargs) >>>>          out, err = process.communicate() >>>> >>>>          if not bytes_output: >>>>              out = out.decode("utf-8") >>>>              err = err.decode("utf-8") >>>> >>>>          if process.returncode: >>>>>            raise ShellError(process.returncode, out, err) >>>> E           ost_utils.shell.ShellError: Command failed with rc=1. >>>> Stdout: >>>> E >>>> E           Stderr: >>>> E           unable to load certificate >>>> E           139734854465344:error:0909006C:PEM routines:get_name:no >>>> start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE >>> (As I said, didn't check myself - I suppose that hsts causes httpd to >>> return some kind of redirect, and this is the way openssl fails when >>> we input this redirect instead of a cert). >>> >>> Best regards, >>> >