- Devel - oVirt List Archives

podman-remote finally works in CI for UI tests
by Marcin Sobczyk 01 Mar '21

01 Mar '21

Hi all, with latest update of podman we should finally able to run UI tests in CI again. I posted a WIP patch for OST that should make this happen, please see the commit message for details: https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113715/ Ehud, Galit, could you please try to adapt the global setup process, so that it runs podman system service? Regards, Marcin

2 1

virsh on compute host
by Paul Dyer 27 Feb '21

27 Feb '21

I am trying to use the command line to stop and start vms in ovirt 4.4. Previously, I used virsh on the standalone engine machine, but that has been removed in this release. On the compute host, I am able to use virsh to list vms, and shutdown vms from that list. The start command gives me this error: virsh # start r7-dante error: failed to get domain 'r7-dante' I have also tried to use curl with the restapi from the standalone engine. I have not been able to find examples of how to stop and start vms. Here is an example of my list of vms... curl --cacert '/etc/pki/ovirt-engine/ctg.crt' --request GET --header 'Version: 4' --header 'Accept: application/xml' --user 'admin@internal:psw' https://bacchus.neworleans.com/ovirt-engine/api/vms Can anyone help me with this? Thank you. Regards -- Paul Dyer, Mercury Consulting Group, RHCE 504-338-8750

2 3

ovirt-engine has been tagged (ovirt-engine-4.4.5.7)
by Tal Nisan 24 Feb '21

24 Feb '21

1 0

Publish master more often
by Yedidyah Bar David 24 Feb '21

24 Feb '21

Hi all, Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node. If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent). So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think. One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org) Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours. This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often. I think I prefer the latter. What do you think? Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/ [2] https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x8… [3] https://jenkins.ovirt.org/job/ost-images_master_standard-poll-upstream-sour… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ -- Didi

4 5

test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)
by Yedidyah Bar David 23 Feb '21

23 Feb '21

On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > Build Number: 894 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #894 > [Andrej Cernek] ost_utils: Remove explicit object inheritance > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate] > > Error Message: > ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > > Stack Trace: > key_format = 'X509-PEM-CA' > verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' > engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0> > > @pytest.mark.parametrize("key_format, verification_fn", [ > pytest.param( > 'X509-PEM-CA', > lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), > id="CA certificate" > ), > pytest.param( > 'OPENSSH-PUBKEY', > lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), > id="ssh pubkey" > ), > ]) > @order_by(_TEST_LIST) > def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, > engine_download): > url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}' I guess (didn't check, only looked at engine git log) that this is a result of [1]. Anyone looking at this? This is trying to download the engine ca cert via http, and then do some verification on it. Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it. For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts. [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 > > with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: > engine_download(url.format(engine_fqdn, key_format), tmp.name) > try: > > verification_fn(tmp.name) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> > lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] > bytes_output = False, kwargs = {} > process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n' > > def shell(args, bytes_output=False, **kwargs): > process = subprocess.Popen(args, > stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > **kwargs) > out, err = process.communicate() > > if not bytes_output: > out = out.decode("utf-8") > err = err.decode("utf-8") > > if process.returncode: > > raise ShellError(process.returncode, out, err) > E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > E > E Stderr: > E unable to load certificate > E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE (As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert). Best regards, -- Didi

4 8

test_incremental_backup_vm2 failed (was: Change in ovirt-system-tests[master]: WIP: Unite basic-suite-master 002_ and 004_ with HE)
by Yedidyah Bar David 18 Feb '21

18 Feb '21

Hi all, On Thu, Feb 18, 2021 at 11:17 AM Code Review <gerrit(a)ovirt.org> wrote: > > From Jenkins CI <jenkins(a)ovirt.org>: > > Jenkins CI has posted comments on this change. ( https://gerrit.ovirt.org/c/ovirt-system-tests/+/113452 ) This patch ^^^ makes 002_ and 004_ test modules identical between basic-suite and he-basic-suite. > > Change subject: WIP: Unite basic-suite-master 002_ and 004_ with HE > ...................................................................... > > > Patch Set 29: Continuous-Integration-1 > > Build Failed > > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15615/ : FAILURE CI ran 3 suites on it. basic-suite and ansible-suite passed, he-basic-suite failed [1] with this in engine.log [2]: ======================================================================== 2021-02-18 10:06:33,389+01 INFO [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [test_incremental_backup] Exception in invoking callback of command RedefineVmCheckpoint (20051278-ed39-40e5-9350-c47bd51fd6c0): ClassCastException: class org.ovirt.engine.core.bll.RedefineVmCheckpointCommand cannot be cast to class org.ovirt.engine.core.bll.SerialChildExecutingCommand (org.ovirt.engine.core.bll.RedefineVmCheckpointCommand and org.ovirt.engine.core.bll.SerialChildExecutingCommand are in unnamed module of loader 'deployment.engine.ear.bll.jar' @176ed0f7) 2021-02-18 10:06:33,389+01 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [test_incremental_backup] Error invoking callback method 'doPolling' for 'ACTIVE' command '20051278-ed39-40e5-9350-c47bd51fd6c0' 2021-02-18 10:06:33,390+01 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [test_incremental_backup] Exception: java.lang.ClassCastException: class org.ovirt.engine.core.bll.RedefineVmCheckpointCommand cannot be cast to class org.ovirt.engine.core.bll.SerialChildExecutingCommand (org.ovirt.engine.core.bll.RedefineVmCheckpointCommand and org.ovirt.engine.core.bll.SerialChildExecutingCommand are in unnamed module of loader 'deployment.engine.ear.bll.jar' @176ed0f7) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:29) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) ======================================================================== Any idea? I also now see that basic-suite fails [3], but on a different test - test_import_floating_disk. Not sure that's related. Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15615… [2] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15615… [3] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > > > -- > To view, visit https://gerrit.ovirt.org/c/ovirt-system-tests/+/113452 > To unsubscribe, or for help writing mail filters, visit https://gerrit.ovirt.org/settings > > Gerrit-Project: ovirt-system-tests > Gerrit-Branch: master > Gerrit-Change-Id: Ied836cda6b622dbebdb869e0b83fa4c0e0b7ca2c > Gerrit-Change-Number: 113452 > Gerrit-PatchSet: 29 > Gerrit-Owner: Yedidyah Bar David <didi(a)redhat.com> > Gerrit-Reviewer: Anton Marchukov <amarchuk(a)redhat.com> > Gerrit-Reviewer: Dafna Ron <dron(a)redhat.com> > Gerrit-Reviewer: Dusan Fodor <dfodor(a)redhat.com> > Gerrit-Reviewer: Gal Ben Haim <galbh2(a)gmail.com> > Gerrit-Reviewer: Galit Rosenthal <grosenth(a)redhat.com> > Gerrit-Reviewer: Jenkins CI <jenkins(a)ovirt.org> > Gerrit-Reviewer: Name of user not set #1001916 > Gerrit-Reviewer: Yedidyah Bar David <didi(a)redhat.com> > Gerrit-Reviewer: Zuul CI <zuul(a)ovirt.org> > Gerrit-Comment-Date: Thu, 18 Feb 2021 09:17:18 +0000 > Gerrit-HasComments: No > Gerrit-Has-Labels: Yes > Gerrit-MessageType: comment > -- Didi

2 1

ovirt-engine has been tagged (ovirt-engine-4.4.5.6)
by Tal Nisan 17 Feb '21

17 Feb '21

2 1

Support for SSH keys other than RSA
by Artur Socha 12 Feb '21

12 Feb '21

Hi, I have been recently working on adding support for SSH keys other than RSA (communication between ovirt-engine and hosts(VDS-es)). The entire effort is tracked in Bugzilla [1]. There are couple important changes I would like to share with you. First and the most important is changing the way connection is verified. Previously fingerprints (by default SHA-256 unless changed via configuration) were used to verify if the connection between the engine and the host could be established. Now public keys are compared instead (with one exception for backward compatibility). For backward compatibility ie. for previously added (legacy) hosts with fingerprint calculated out of RSA public key (the key not stored in db) the verification is done as before that means we compare fingerprints only. After upgrade the whole setup is expected to work without any manual intervention. However, there are couple of options to 'migrate' legacy fingerprint to whatever ssh server finds the strongest on the host: 1) In database remove sshkeyfingerprint value ie. update vds_static set sshkeyfingerprint='' where vds_id = 'PUT_HERE_HOST_ID' 2) REST:prepare request with blank fingerprint for 'legacy' hosts Please see the (documentation [2]). Fingerprint and public key will be re-entered, 3) reinstall host / install new host 4) manually deploy key and update host's VDS_static.sshkeyfingerprint and vds_static.public_key On engine's UI side there is still a way to fetch fingerprints (on 'New Host' panel but we anticipate that soon there will be a public key (open ssh format) instead. Please let me know if you have any questions, doubts or if you encounter any issues around this area. Patches (referenced in BZ[1]) has been merged into master and this feature is expected to go with 4.4.5 upstream release. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1837221 [2] https://jenkins.ovirt.org/job/ovirt-engine-api-model_standard-check-patch/1… best, Artur

2 1

ansible: Unsupported parameters found in auth
by Yedidyah Bar David 12 Feb '21

12 Feb '21

Hi all, he-basic-suite fails [1] with: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Ensure that the target datacenter is present] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unsupported parameters for (ovirt_datacenter) module: compress, timeout found in auth. Supported parameters include: ca_file, headers, hostname, insecure, kerberos, password, token, url, username"} Any clue? Thanks, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1918/ -- Didi

2 1

ovirt-engine has been tagged (ovirt-engine-4.4.5.5)
by Tal Nisan 10 Feb '21

10 Feb '21

2 1