[ovirt-devel] Re: test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)

On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_night... > Build Number: 894 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #894 > [Andrej Cernek] ost_utils: Remove explicit object inheritance > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate] > > Error Message: > ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > > Stack Trace: > key_format = 'X509-PEM-CA' > verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' > engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0> > > @pytest.mark.parametrize("key_format, verification_fn", [ > pytest.param( > 'X509-PEM-CA', > lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), > id="CA certificate" > ), > pytest.param( > 'OPENSSH-PUBKEY', > lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), > id="ssh pubkey" > ), > ]) > @order_by(_TEST_LIST) > def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, > engine_download): > url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}' I guess (didn't check, only looked at engine git log) that this is a result of [1]. Anyone looking at this? This is trying to download the engine ca cert via http, and then do some verification on it. Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it. For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts. [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 > > with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: > engine_download(url.format(engine_fqdn, key_format), tmp.name) > try: >> verification_fn(tmp.name) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> > lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] > bytes_output = False, kwargs = {} > process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n' > > def shell(args, bytes_output=False, **kwargs): > process = subprocess.Popen(args, > stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > **kwargs) > out, err = process.communicate() > > if not bytes_output: > out = out.decode("utf-8") > err = err.decode("utf-8") > > if process.returncode: >> raise ShellError(process.returncode, out, err) > E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > E > E Stderr: > E unable to load certificate > E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE (As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert). Best regards,