test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 894 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #894 [Andrej Cernek] ost_utils: Remove explicit object inheritance
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate]
Error Message: ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Stack Trace: key_format = 'X509-PEM-CA' verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0>
@pytest.mark.parametrize("key_format, verification_fn", [ pytest.param( 'X509-PEM-CA', lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), id="CA certificate" ), pytest.param( 'OPENSSH-PUBKEY', lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), id="ssh pubkey" ), ]) @order_by(_TEST_LIST) def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, engine_download): url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}'
I guess (didn't check, only looked at engine git log) that this is a result of [1]. Anyone looking at this? This is trying to download the engine ca cert via http, and then do some verification on it. Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it. For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts. [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: engine_download(url.format(engine_fqdn, key_format), tmp.name) try:
verification_fn(tmp.name)
../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n'
def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate()
if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8")
if process.returncode:
raise ShellError(process.returncode, out, err)
E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: E E Stderr: E unable to load certificate E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert). Best regards, -- Didi
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs. Artur On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 894 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #894 [Andrej Cernek] ost_utils: Remove explicit object inheritance
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate]
Error Message: ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Stack Trace: key_format = 'X509-PEM-CA' verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0>
@pytest.mark.parametrize("key_format, verification_fn", [ pytest.param( 'X509-PEM-CA', lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), id="CA certificate" ), pytest.param( 'OPENSSH-PUBKEY', lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), id="ssh pubkey" ), ]) @order_by(_TEST_LIST) def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, engine_download): url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}'
I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: engine_download(url.format(engine_fqdn, key_format), tmp.name) try:
verification_fn(tmp.name)
../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n'
def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate()
if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8")
if process.returncode:
raise ShellError(process.returncode, out, err)
E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: E E Stderr: E unable to load certificate E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com> wrote:
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs.
If we already have an exception, and hopefully some process to add one, then I think it makes sense for this case as well. I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means. This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should. TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510 Best regards,
Artur
On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 894 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #894 [Andrej Cernek] ost_utils: Remove explicit object inheritance
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate]
Error Message: ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Stack Trace: key_format = 'X509-PEM-CA' verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0>
@pytest.mark.parametrize("key_format, verification_fn", [ pytest.param( 'X509-PEM-CA', lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), id="CA certificate" ), pytest.param( 'OPENSSH-PUBKEY', lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), id="ssh pubkey" ), ]) @order_by(_TEST_LIST) def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, engine_download): url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}'
I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: engine_download(url.format(engine_fqdn, key_format), tmp.name) try:
verification_fn(tmp.name)
../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n'
def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate()
if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8")
if process.returncode:
raise ShellError(process.returncode, out, err)
E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: E E Stderr: E unable to load certificate E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
-- Didi
Hi, On 2/22/21 4:21 PM, Yedidyah Bar David wrote:
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs. If we already have an exception, and hopefully some process to add one,
On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com> wrote: then I think it makes sense for this case as well.
I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means.
This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should.
TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510
Best regards, For now I posted a patch for OST that will unblock basic suite [2]. When we have a proper solution we should adapt the tests to the new way of working.
Regards, Marcin [2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/
Artur
On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/... Build Number: 894 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #894 [Andrej Cernek] ost_utils: Remove explicit object inheritance
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate]
Error Message: ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Stack Trace: key_format = 'X509-PEM-CA' verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0>
@pytest.mark.parametrize("key_format, verification_fn", [ pytest.param( 'X509-PEM-CA', lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), id="CA certificate" ), pytest.param( 'OPENSSH-PUBKEY', lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), id="ssh pubkey" ), ]) @order_by(_TEST_LIST) def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, engine_download): url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}' I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: engine_download(url.format(engine_fqdn, key_format), tmp.name) try:
verification_fn(tmp.name)
../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n'
def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate()
if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8")
if process.returncode:
raise ShellError(process.returncode, out, err)
E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: E E Stderr: E unable to load certificate E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
And the fix for the engine is here: https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/ Artur On 22.02.2021 16:29, Marcin Sobczyk wrote:
Hi,
On 2/22/21 4:21 PM, Yedidyah Bar David wrote:
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs. If we already have an exception, and hopefully some process to add one,
On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com> wrote: then I think it makes sense for this case as well.
I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means.
This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should.
TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510
Best regards, For now I posted a patch for OST that will unblock basic suite [2]. When we have a proper solution we should adapt the tests to the new way of working.
Regards, Marcin
[2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/
Artur
On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/...
Build Number: 894 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #894 [Andrej Cernek] ost_utils: Remove explicit object inheritance
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate]
Error Message: ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Stack Trace: key_format = 'X509-PEM-CA' verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0>
@pytest.mark.parametrize("key_format, verification_fn", [ pytest.param( 'X509-PEM-CA', lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), id="CA certificate" ), pytest.param( 'OPENSSH-PUBKEY', lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), id="ssh pubkey" ), ]) @order_by(_TEST_LIST) def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, engine_download): url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}'
I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: engine_download(url.format(engine_fqdn, key_format), tmp.name) try:
verification_fn(tmp.name) ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292:
_ _ _ _ _ _ ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n'
def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate()
if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8")
if process.returncode:
raise ShellError(process.returncode, out, err) E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: E E Stderr: E unable to load certificate E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE (As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
Given the code freeze this week, could you please merge ASAP, so that we can run OST with other patches? Thanks Vojta On Monday, 22 February 2021 17:07:49 CET Artur Socha wrote:
And the fix for the engine is here: https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/
Artur
On 22.02.2021 16:29, Marcin Sobczyk wrote:
Hi,
On 2/22/21 4:21 PM, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com> wrote:
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs.
If we already have an exception, and hopefully some process to add one, then I think it makes sense for this case as well.
I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means.
This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should.
TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510
Best regards,
For now I posted a patch for OST that will unblock basic suite [2]. When we have a proper solution we should adapt the tests to the new way of working.
Regards, Marcin
[2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/
Artur
On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote:
Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni ghtly/
Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni ghtly/894/
Build Number: 894 Build Status: Failure Triggered By: Started by timer
------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #894 [Andrej Cernek] ost_utils: Remove explicit object inheritance
----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engin e_certs[CA certificate]
Error Message: ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Stack Trace: key_format = 'X509-PEM-CA' verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0>
@pytest.mark.parametrize("key_format, verification_fn", [ pytest.param( 'X509-PEM-CA', lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), id="CA certificate" ), pytest.param( 'OPENSSH-PUBKEY', lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), id="ssh pubkey" ), ]) @order_by(_TEST_LIST) def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, engine_download): url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate& format={}'>>>> I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: engine_download(url.format(engine_fqdn, key_format), tmp.name)
try: > verification_fn(tmp.name)
../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] bytes_output = False, kwargs = {} process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n'
def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate()
if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8")
if process.returncode: > raise ShellError(process.returncode, out, err)
E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: E E Stderr: E unable to load certificate E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
On Tue, Feb 23, 2021 at 9:42 AM Vojtech Juranek <vjuranek@redhat.com> wrote:
Given the code freeze this week, could you please merge ASAP, so that we can run OST with other patches?
As I commented on the bug (1926625), it seems like hosted-engine will require more work. Considering the planned code freeze, I suggest to revert the original patch for now. Best regards,
Thanks Vojta
On Monday, 22 February 2021 17:07:49 CET Artur Socha wrote:
And the fix for the engine is here: https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/
Artur
On 22.02.2021 16:29, Marcin Sobczyk wrote:
Hi,
On 2/22/21 4:21 PM, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com> wrote:
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs.
If we already have an exception, and hopefully some process to add one, then I think it makes sense for this case as well.
I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means.
This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should.
TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510
Best regards,
For now I posted a patch for OST that will unblock basic suite [2]. When we have a proper solution we should adapt the tests to the new way of working.
Regards, Marcin
[2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/
Artur
On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote: > Project: > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni > ghtly/ > > Build: > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni > ghtly/894/ > > Build Number: 894 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #894 > [Andrej Cernek] ost_utils: Remove explicit object inheritance > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: > basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engin > e_certs[CA certificate] > > Error Message: > ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > Stderr: unable to load certificate > 139734854465344:error:0909006C:PEM routines:get_name:no start > line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > > Stack Trace: > key_format = 'X509-PEM-CA' > verification_fn = <function <lambda> at 0x7f6aab2add90>, > engine_fqdn = 'engine' > engine_download = <function engine_download.<locals>.download at > 0x7f6aa98d5ea0> > > @pytest.mark.parametrize("key_format, verification_fn", [ > pytest.param( > 'X509-PEM-CA', > lambda path: shell.shell(["openssl", "x509", "-in", > path, "-text", "-noout"]), > id="CA certificate" > ), > pytest.param( > 'OPENSSH-PUBKEY', > lambda path: shell.shell(["ssh-keygen", "-l", "-f", > path]), > id="ssh pubkey" > ), > ]) > @order_by(_TEST_LIST) > def test_verify_engine_certs(key_format, verification_fn, > engine_fqdn, > engine_download): > url = > 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate& > format={}'>>>> I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
> with http_proxy_disabled(), tempfile.NamedTemporaryFile() > as tmp: > engine_download(url.format(engine_fqdn, key_format), > tmp.name) > > try: >> verification_fn(tmp.name) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in > <lambda> > lambda path: shell.shell(["openssl", "x509", "-in", path, > "-text", "-noout"]), > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ > > args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', > '-noout'] > bytes_output = False, kwargs = {} > process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > err = 'unable to load > certificate\n139734854465344:error:0909006C:PEM > routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: > TRUSTED CERTIFICATE\n' > > def shell(args, bytes_output=False, **kwargs): > process = subprocess.Popen(args, > stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > **kwargs) > out, err = process.communicate() > > if not bytes_output: > out = out.decode("utf-8") > err = err.decode("utf-8") > > if process.returncode: >> raise ShellError(process.returncode, out, err) > > E ost_utils.shell.ShellError: Command failed with rc=1. > Stdout: > E > E Stderr: > E unable to load certificate > E 139734854465344:error:0909006C:PEM routines:get_name:no > start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/N72N67VDSY2Z55...
-- Didi
In case of the fix for the engine I still cannot verify it because it fails when I run OST locally, I am leaning towards reverting the offending patch. Artur On Tue, Feb 23, 2021 at 8:42 AM Vojtech Juranek <vjuranek@redhat.com> wrote:
Given the code freeze this week, could you please merge ASAP, so that we can run OST with other patches? Thanks Vojta
And the fix for the engine is here: https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/
Artur
On 22.02.2021 16:29, Marcin Sobczyk wrote:
Hi,
On 2/22/21 4:21 PM, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com> wrote:
Hi Didi, You are probably right that enabling Strict Transport Security caused that bug as an unfortunate side-effect. Do you think that, adding some sort of exception for cert url would be an acceptable fix? For example we have this kind of rule for excluding authentication for Rest api docs.
If we already have an exception, and hopefully some process to add one, then I think it makes sense for this case as well.
I admit, though, that I do not feel completely happy with this. On one hand, this is insecure, and on the other hand, there is no way to do this securely using the existing official means.
This thread also made me think about the hosted-engine deploy process. In standalone engine setup, the user is responsible for installing the OS, so it's up to the user to control (or not) generation of the sshd private key for allowing later secure access to it using ssh. For hosted-engine, it's us, and I do not think we do anything around this. Perhaps we should.
TL;DR: IMO: 1. Please add an exception. Please open another bug for this. 2. We should document how to get the engine CA cert not using https: ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . 3. We should consider our options for hosted-engine. Filed now [1].
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510
Best regards,
For now I posted a patch for OST that will unblock basic suite [2]. When we have a proper solution we should adapt the tests to the new way of working.
Regards, Marcin
[2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/
Artur
On 22.02.2021 13:52, Yedidyah Bar David wrote:
On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org> wrote: > Project: > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni > ghtly/ > > Build: > https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni > ghtly/894/ > > Build Number: 894 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #894 > [Andrej Cernek] ost_utils: Remove explicit object inheritance > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: > basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engin > e_certs[CA certificate] > > Error Message: > ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > Stderr: unable to load certificate > 139734854465344:error:0909006C:PEM routines:get_name:no start > line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > > Stack Trace: > key_format = 'X509-PEM-CA' > verification_fn = <function <lambda> at 0x7f6aab2add90>, > engine_fqdn = 'engine' > engine_download = <function engine_download.<locals>.download at > 0x7f6aa98d5ea0> > > @pytest.mark.parametrize("key_format, verification_fn", [ > pytest.param( > 'X509-PEM-CA', > lambda path: shell.shell(["openssl", "x509", "-in", > path, "-text", "-noout"]), > id="CA certificate" > ), > pytest.param( > 'OPENSSH-PUBKEY', > lambda path: shell.shell(["ssh-keygen", "-l", "-f", > path]), > id="ssh pubkey" > ), > ]) > @order_by(_TEST_LIST) > def test_verify_engine_certs(key_format, verification_fn, > engine_fqdn, > engine_download): > url = > 'http:// {}/ovirt-engine/services/pki-resource?resource=ca-certificate& > format={}'>>>> I guess (didn't check, only looked at engine git log) that this is a result of [1].
Anyone looking at this?
This is trying to download the engine ca cert via http, and then do some verification on it.
Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it.
For OST, it might be easy to fix by s/http/https/ and perhaps
some param to make it not check certs in https. But I find it quite reasonable
others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix
On Monday, 22 February 2021 17:07:49 CET Artur Socha wrote: passing that their
stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts.
[1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508
> with http_proxy_disabled(), tempfile.NamedTemporaryFile() > as tmp: > engine_download(url.format(engine_fqdn, key_format), > tmp.name) > > try: >> verification_fn(tmp.name) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in > <lambda> > lambda path: shell.shell(["openssl", "x509", "-in", path, > "-text", "-noout"]), > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ _ _ _ _ _ > > args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', > '-noout'] > bytes_output = False, kwargs = {} > process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > err = 'unable to load > certificate\n139734854465344:error:0909006C:PEM > routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: > TRUSTED CERTIFICATE\n' > > def shell(args, bytes_output=False, **kwargs): > process = subprocess.Popen(args, > stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > **kwargs) > out, err = process.communicate() > > if not bytes_output: > out = out.decode("utf-8") > err = err.decode("utf-8") > > if process.returncode: >> raise ShellError(process.returncode, out, err) > > E ost_utils.shell.ShellError: Command failed with rc=1. > Stdout: > E > E Stderr: > E unable to load certificate > E 139734854465344:error:0909006C:PEM routines:get_name:no > start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
(As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert).
Best regards,
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/N72N67VDSY2Z55...
-- Artur Socha Senior Software Engineer, RHV Red Hat
The revert patch is here: https://gerrit.ovirt.org/#/c/ovirt-engine/+/113659/ Artur On 23.02.2021 08:56, Artur Socha wrote:
In case of the fix for the engine I still cannot verify it because it fails when I run OST locally, I am leaning towards reverting the offending patch. Artur
On Tue, Feb 23, 2021 at 8:42 AM Vojtech Juranek <vjuranek@redhat.com <mailto:vjuranek@redhat.com>> wrote:
Given the code freeze this week, could you please merge ASAP, so that we can run OST with other patches? Thanks Vojta
On Monday, 22 February 2021 17:07:49 CET Artur Socha wrote: > And the fix for the engine is here: > https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/ <https://gerrit.ovirt.org/#/c/ovirt-engine/+/113650/> > > Artur > > On 22.02.2021 16:29, Marcin Sobczyk wrote: > > Hi, > > > > On 2/22/21 4:21 PM, Yedidyah Bar David wrote: > >> On Mon, Feb 22, 2021 at 4:51 PM Artur Socha <asocha@redhat.com <mailto:asocha@redhat.com>> wrote: > >>> Hi Didi, > >>> You are probably right that enabling Strict Transport Security caused > >>> that bug as an unfortunate side-effect. > >>> Do you think that, adding some sort of exception for cert url would be > >>> an acceptable fix? For example we have this kind of rule for excluding > >>> authentication for Rest api docs. > >> > >> If we already have an exception, and hopefully some process to add one, > >> then I think it makes sense for this case as well. > >> > >> I admit, though, that I do not feel completely happy with this. On one > >> hand, > >> this is insecure, and on the other hand, there is no way to do this > >> securely > >> using the existing official means. > >> > >> This thread also made me think about the hosted-engine deploy process. > >> In standalone engine setup, the user is responsible for installing the > >> OS, > >> so it's up to the user to control (or not) generation of the sshd > >> private key > >> for allowing later secure access to it using ssh. For hosted-engine, > >> it's us, > >> and I do not think we do anything around this. Perhaps we should. > >> > >> TL;DR: IMO: > >> 1. Please add an exception. Please open another bug for this. > >> 2. We should document how to get the engine CA cert not using https: > >> ssh to the engine machine; cat /etc/pki/ovirt-engine/ca.pem . > >> 3. We should consider our options for hosted-engine. Filed now [1]. > >> > >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1931510 <https://bugzilla.redhat.com/show_bug.cgi?id=1931510> > >> > >> Best regards, > > > > For now I posted a patch for OST that will unblock basic suite [2]. > > When we have a proper solution we should adapt the tests to the new way > > of working. > > > > Regards, Marcin > > > > [2] https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/ <https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113649/> > > > >>> Artur > >>> > >>> On 22.02.2021 13:52, Yedidyah Bar David wrote: > >>>> On Mon, Feb 22, 2021 at 3:12 AM <jenkins@jenkins.phx.ovirt.org <mailto:jenkins@jenkins.phx.ovirt.org>> wrote: > >>>>> Project: > >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni <https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni> > >>>>> ghtly/ > >>>>> > >>>>> Build: > >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni <https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_ni> > >>>>> ghtly/894/ > >>>>> > >>>>> Build Number: 894 > >>>>> Build Status: Failure > >>>>> Triggered By: Started by timer > >>>>> > >>>>> ------------------------------------- > >>>>> Changes Since Last Success: > >>>>> ------------------------------------- > >>>>> Changes for Build #894 > >>>>> [Andrej Cernek] ost_utils: Remove explicit object inheritance > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> ----------------- > >>>>> Failed Tests: > >>>>> ----------------- > >>>>> 1 tests failed. > >>>>> FAILED: > >>>>> basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engin > >>>>> e_certs[CA certificate] > >>>>> > >>>>> Error Message: > >>>>> ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > >>>>> Stderr: unable to load certificate > >>>>> 139734854465344:error:0909006C:PEM routines:get_name:no start > >>>>> line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > >>>>> > >>>>> Stack Trace: > >>>>> key_format = 'X509-PEM-CA' > >>>>> verification_fn = <function <lambda> at 0x7f6aab2add90>, > >>>>> engine_fqdn = 'engine' > >>>>> engine_download = <function engine_download.<locals>.download at > >>>>> 0x7f6aa98d5ea0> > >>>>> > >>>>> @pytest.mark.parametrize("key_format, verification_fn", [ > >>>>> pytest.param( > >>>>> 'X509-PEM-CA', > >>>>> lambda path: shell.shell(["openssl", "x509", "-in", > >>>>> path, "-text", "-noout"]), > >>>>> id="CA certificate" > >>>>> ), > >>>>> pytest.param( > >>>>> 'OPENSSH-PUBKEY', > >>>>> lambda path: shell.shell(["ssh-keygen", "-l", "-f", > >>>>> path]), > >>>>> id="ssh pubkey" > >>>>> ), > >>>>> ]) > >>>>> @order_by(_TEST_LIST) > >>>>> def test_verify_engine_certs(key_format, verification_fn, > >>>>> engine_fqdn, > >>>>> engine_download): > >>>>> url = > >>>>> 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate& > >>>>> format={}'>>>> > >>>> I guess (didn't check, only looked at engine git log) that this is a > >>>> result of [1]. > >>>> > >>>> Anyone looking at this? > >>>> > >>>> This is trying to download the engine ca cert via http, and then do > >>>> some verification on it. > >>>> > >>>> Generally speaking, this is a chicken-and-egg problem: You can't > >>>> securely download > >>>> a ca cert if you need this cert to securely download it. > >>>> > >>>> For OST, it might be easy to fix by s/http/https/ and perhaps passing > >>>> some param to > >>>> make it not check certs in https. But I find it quite reasonable that > >>>> others are doing > >>>> similar things and will now be broken by this change [1]. If so, we > >>>> might decide that > >>>> this is "by design" - that whoever that gets broken, should fix their > >>>> stuff one way or > >>>> another (like OST above, or via safer means if possible/relevant, such > >>>> as using ssh > >>>> to securely connect to the engine machine and then get the cert from > >>>> there somehow > >>>> (do we have an api for this?)). Or we can decide that it's an engine > >>>> bug - that [1] > >>>> should have allowed this specific url to bypass hsts. > >>>> > >>>> [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 <https://gerrit.ovirt.org/c/ovirt-engine/+/113508> > >>>> > >>>>> with http_proxy_disabled(), tempfile.NamedTemporaryFile() > >>>>> as tmp: > >>>>> engine_download(url.format(engine_fqdn, key_format), > >>>>> tmp.name <http://tmp.name>) > >>>>> > >>>>> try: > >>>>>> verification_fn(tmp.name <http://tmp.name>) > >>>>> > >>>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >>>>> _ _ _ _ _ _ > >>>>> ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in > >>>>> <lambda> > >>>>> lambda path: shell.shell(["openssl", "x509", "-in", path, > >>>>> "-text", "-noout"]), > >>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > >>>>> _ _ _ _ _ _ > >>>>> > >>>>> args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', > >>>>> '-noout'] > >>>>> bytes_output = False, kwargs = {} > >>>>> process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > >>>>> err = 'unable to load > >>>>> certificate\n139734854465344:error:0909006C:PEM > >>>>> routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: > >>>>> TRUSTED CERTIFICATE\n' > >>>>> > >>>>> def shell(args, bytes_output=False, **kwargs): > >>>>> process = subprocess.Popen(args, > >>>>> stdout=subprocess.PIPE, > >>>>> stderr=subprocess.PIPE, > >>>>> **kwargs) > >>>>> out, err = process.communicate() > >>>>> > >>>>> if not bytes_output: > >>>>> out = out.decode("utf-8") > >>>>> err = err.decode("utf-8") > >>>>> > >>>>> if process.returncode: > >>>>>> raise ShellError(process.returncode, out, err) > >>>>> > >>>>> E ost_utils.shell.ShellError: Command failed with rc=1. > >>>>> Stdout: > >>>>> E > >>>>> E Stderr: > >>>>> E unable to load certificate > >>>>> E 139734854465344:error:0909006C:PEM routines:get_name:no > >>>>> start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > >>>> > >>>> (As I said, didn't check myself - I suppose that hsts causes httpd to > >>>> return some kind of redirect, and this is the way openssl fails when > >>>> we input this redirect instead of a cert). > >>>> > >>>> Best regards,
_______________________________________________ Devel mailing list -- devel@ovirt.org <mailto:devel@ovirt.org> To unsubscribe send an email to devel-leave@ovirt.org <mailto:devel-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/N72N67VDSY2Z55... <https://lists.ovirt.org/archives/list/devel@ovirt.org/message/N72N67VDSY2Z55WQOSW2Y24ZBB3KGARS/>
-- Artur Socha Senior Software Engineer, RHV Red Hat
participants (4)
-
Artur Socha -
Marcin Sobczyk -
Vojtech Juranek -
Yedidyah Bar David