February 2021 - Devel - oVirt List Archives

[VDSM] Started getting virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired
by Edward Haas 11 Jan '24

11 Jan '24

Hi, I have a VM which acts as an oVirt host with VDSM installed. I use it for testing without connecting it to Engine. Recently VDSM fails to come up and I see this error: Sep 26 17:58:18 localhost libvirtd: 2018-09-26 14:58:18.978+0000: 12176: error : virNetTLSContextCheckCertTimes:154 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem has expired Any hint on how to generate or fetch a new certificate without the help of Engine? Thanks, Edy.

3 5

Error: Adding new Host to ovirt-engine
by Ahmad Khiet 14 Aug '23

14 Aug '23

Hi, Can't add new host to ovirt engine, because the following error: 2019-06-12 12:23:09,664 p=4134 u=engine | TASK [ovirt-host-deploy-facts : Set facts] ************************************* 2019-06-12 12:23:09,684 p=4134 u=engine | ok: [10.35.1.17] => { "ansible_facts": { "ansible_python_interpreter": "/usr/bin/python2", "host_deploy_vdsm_version": "4.40.0" }, "changed": false } 2019-06-12 12:23:09,697 p=4134 u=engine | TASK [ovirt-provider-ovn-driver : Install ovs] ********************************* 2019-06-12 12:23:09,726 p=4134 u=engine | fatal: [10.35.1.17]: FAILED! => {} MSG: The conditional check 'cluster_switch == "ovs" or (ovn_central is defined and ovn_central | ipaddr and ovn_engine_cluster_version is version_compare('4.2', '>='))' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller The error appears to be in '/home/engine/apps/engine/share/ovirt-engine/playbooks/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - block: - name: Install ovs ^ here 2019-06-12 12:23:09,728 p=4134 u=engine | PLAY RECAP ********************************************************************* 2019-06-12 12:23:09,728 p=4134 u=engine | 10.35.1.17 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 whats missing!? Thanks -- Ahmad Khiet Red Hat <https://www.redhat.com/> akhiet(a)redhat.com M: +972-54-6225629 <https://red.ht/sig>

7 10

Purging inactive maintainers from vdsm-master-maintainers
by Nir Soffer 21 Jun '21

21 Jun '21

I want to remove inactive contributors from vdsm-master-maintainers. I suggest the simple rule of 2 years of inactivity for removing from this group, based on git log. See the list below for current status: https://gerrit.ovirt.org/#/admin/groups/106,members Thoughts? Nir

6 11

Merge rights changes in the oVirt Engine project
by Tal Nisan 09 Apr '21

09 Apr '21

Hi everyone, As you probably know we are now in a mode in which we develop our next zstream version on the master branch as opposed to how we worked before where the master version was dedicated for the next major version. This makes the rapid changes in master to be delivered to customers in a much higher cadence thus affecting stability. Due to that we think it's best that from now on merges in the master branch will be done only by stable branch maintainers after inspecting those closely. What you need to do in order to get your patch merged: - Have it pass Jenkins - Have it get code review +2 - Have it mark verified +1 - It's always encourage to have it tested by OST, for bigger changes it's a must Once you have all those covered, please add me as a reviewer and I'll examine it and merge if everything seems right, if I haven't done it in a timely manner feel free to ping me.

3 4

Error Java SDK Issue??
by Geschwentner, Patrick 02 Apr '21

02 Apr '21

Dear Ladies and Gentlemen! I am currently working with the java-sdk and I encountered a problem. If I would like to retrieve the disk details, I get the following error: Disk currDisk = ovirtConnection.followLink(diskAttachment.disk()); The Error is occurring in this line: [cid:image001.png@01D44537.AF127FD0] The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok). Error: wrong number of arguments The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ). Can you confirm the defect? Best regards Patrick

6 11

podman-remote finally works in CI for UI tests
by Marcin Sobczyk 01 Mar '21

01 Mar '21

Hi all, with latest update of podman we should finally able to run UI tests in CI again. I posted a WIP patch for OST that should make this happen, please see the commit message for details: https://gerrit.ovirt.org/#/c/ovirt-system-tests/+/113715/ Ehud, Galit, could you please try to adapt the global setup process, so that it runs podman system service? Regards, Marcin

2 1

virsh on compute host
by Paul Dyer 27 Feb '21

27 Feb '21

I am trying to use the command line to stop and start vms in ovirt 4.4. Previously, I used virsh on the standalone engine machine, but that has been removed in this release. On the compute host, I am able to use virsh to list vms, and shutdown vms from that list. The start command gives me this error: virsh # start r7-dante error: failed to get domain 'r7-dante' I have also tried to use curl with the restapi from the standalone engine. I have not been able to find examples of how to stop and start vms. Here is an example of my list of vms... curl --cacert '/etc/pki/ovirt-engine/ctg.crt' --request GET --header 'Version: 4' --header 'Accept: application/xml' --user 'admin@internal:psw' https://bacchus.neworleans.com/ovirt-engine/api/vms Can anyone help me with this? Thank you. Regards -- Paul Dyer, Mercury Consulting Group, RHCE 504-338-8750

2 3

ovirt-engine has been tagged (ovirt-engine-4.4.5.7)
by Tal Nisan 24 Feb '21

24 Feb '21

1 0

Publish master more often
by Yedidyah Bar David 24 Feb '21

24 Feb '21

Hi all, Right now, when we merge a patch e.g. to the engine (and many other projects), it can take up to several days until it is used by the hosted-engine ovirt-system-tests suite. Something similar will happen soon if/when we introduce suites that use ovirt-node. If I got it right: - Merge causes CI to build the engine - immediately, takes ~ 1 hour (say) - A publisher job [1] publishes it to resources.ovirt.org (daily, midnight (UTC)) - The next run of an appliance build [2] includes it (daily, afternoon) - The next run of the publisher [1] publishes the appliance (daily, midnight) - The next run of ost-images [3] includes the appliance (daily, midnight, 2 hours after the publisher) (and publishes it immediately) - The next run of ost (e.g. [4]) will use it (daily, slightly *before* ost-images, but I guess we can change that. And this does not affect manual runs of OST, so can probably be ignored in the calculation, at least to some extent). So if I got it right, a patch merged to the engine in some morning, will be used by the nightly run of OST HE only after almost 3 days, and available for manual runs after 2 days. IMO that's too much time. I might be somewhat wrong, but not very, I think. One partial solution is to add automation .repos lines to relevant projects that will link at lastSuccessfulBuild (let's call it lastSB) of the more important projects they consume - e.g. appliance to use lastSB of engine+dwh+a few others, node to use lastSB of vdsm, etc. This will require more maintenance (adding/removing/fixing projects as needed) and cause some more load on CI (as now packages will be downloaded from it instead of from resources.ovirt.org) Another solution is to run relevant jobs (publisher/appliance/node) far more often - say, once every two hours. This will also add load, and might cause "perceived" instability - as things will likely fluctuate between green and red more often. I think I prefer the latter. What do you think? Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/ [2] https://jenkins.ovirt.org/job/ovirt-appliance_master_build-artifacts-el8-x8… [3] https://jenkins.ovirt.org/job/ost-images_master_standard-poll-upstream-sour… [4] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ -- Didi

4 5

test_verify_engine_certs (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 894 - Failure!)
by Yedidyah Bar David 23 Feb '21

23 Feb '21

On Mon, Feb 22, 2021 at 3:12 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > Build Number: 894 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #894 > [Andrej Cernek] ost_utils: Remove explicit object inheritance > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_verify_engine_certs[CA certificate] > > Error Message: > ost_utils.shell.ShellError: Command failed with rc=1. Stdout: Stderr: unable to load certificate 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE > > Stack Trace: > key_format = 'X509-PEM-CA' > verification_fn = <function <lambda> at 0x7f6aab2add90>, engine_fqdn = 'engine' > engine_download = <function engine_download.<locals>.download at 0x7f6aa98d5ea0> > > @pytest.mark.parametrize("key_format, verification_fn", [ > pytest.param( > 'X509-PEM-CA', > lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), > id="CA certificate" > ), > pytest.param( > 'OPENSSH-PUBKEY', > lambda path: shell.shell(["ssh-keygen", "-l", "-f", path]), > id="ssh pubkey" > ), > ]) > @order_by(_TEST_LIST) > def test_verify_engine_certs(key_format, verification_fn, engine_fqdn, > engine_download): > url = 'http://{}/ovirt-engine/services/pki-resource?resource=ca-certificate&format={}' I guess (didn't check, only looked at engine git log) that this is a result of [1]. Anyone looking at this? This is trying to download the engine ca cert via http, and then do some verification on it. Generally speaking, this is a chicken-and-egg problem: You can't securely download a ca cert if you need this cert to securely download it. For OST, it might be easy to fix by s/http/https/ and perhaps passing some param to make it not check certs in https. But I find it quite reasonable that others are doing similar things and will now be broken by this change [1]. If so, we might decide that this is "by design" - that whoever that gets broken, should fix their stuff one way or another (like OST above, or via safer means if possible/relevant, such as using ssh to securely connect to the engine machine and then get the cert from there somehow (do we have an api for this?)). Or we can decide that it's an engine bug - that [1] should have allowed this specific url to bypass hsts. [1] https://gerrit.ovirt.org/c/ovirt-engine/+/113508 > > with http_proxy_disabled(), tempfile.NamedTemporaryFile() as tmp: > engine_download(url.format(engine_fqdn, key_format), tmp.name) > try: > > verification_fn(tmp.name) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:292: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:275: in <lambda> > lambda path: shell.shell(["openssl", "x509", "-in", path, "-text", "-noout"]), > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > args = ['openssl', 'x509', '-in', '/tmp/tmpnj42cxm2', '-text', '-noout'] > bytes_output = False, kwargs = {} > process = <subprocess.Popen object at 0x7f6aa98143c8>, out = '' > err = 'unable to load certificate\n139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE\n' > > def shell(args, bytes_output=False, **kwargs): > process = subprocess.Popen(args, > stdout=subprocess.PIPE, > stderr=subprocess.PIPE, > **kwargs) > out, err = process.communicate() > > if not bytes_output: > out = out.decode("utf-8") > err = err.decode("utf-8") > > if process.returncode: > > raise ShellError(process.returncode, out, err) > E ost_utils.shell.ShellError: Command failed with rc=1. Stdout: > E > E Stderr: > E unable to load certificate > E 139734854465344:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE (As I said, didn't check myself - I suppose that hsts causes httpd to return some kind of redirect, and this is the way openssl fails when we input this redirect instead of a cert). Best regards, -- Didi

4 8

test_incremental_backup_vm2 failed (was: Change in ovirt-system-tests[master]: WIP: Unite basic-suite-master 002_ and 004_ with HE)
by Yedidyah Bar David 18 Feb '21

18 Feb '21

Hi all, On Thu, Feb 18, 2021 at 11:17 AM Code Review <gerrit(a)ovirt.org> wrote: > > From Jenkins CI <jenkins(a)ovirt.org>: > > Jenkins CI has posted comments on this change. ( https://gerrit.ovirt.org/c/ovirt-system-tests/+/113452 ) This patch ^^^ makes 002_ and 004_ test modules identical between basic-suite and he-basic-suite. > > Change subject: WIP: Unite basic-suite-master 002_ and 004_ with HE > ...................................................................... > > > Patch Set 29: Continuous-Integration-1 > > Build Failed > > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15615/ : FAILURE CI ran 3 suites on it. basic-suite and ansible-suite passed, he-basic-suite failed [1] with this in engine.log [2]: ======================================================================== 2021-02-18 10:06:33,389+01 INFO [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [test_incremental_backup] Exception in invoking callback of command RedefineVmCheckpoint (20051278-ed39-40e5-9350-c47bd51fd6c0): ClassCastException: class org.ovirt.engine.core.bll.RedefineVmCheckpointCommand cannot be cast to class org.ovirt.engine.core.bll.SerialChildExecutingCommand (org.ovirt.engine.core.bll.RedefineVmCheckpointCommand and org.ovirt.engine.core.bll.SerialChildExecutingCommand are in unnamed module of loader 'deployment.engine.ear.bll.jar' @176ed0f7) 2021-02-18 10:06:33,389+01 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [test_incremental_backup] Error invoking callback method 'doPolling' for 'ACTIVE' command '20051278-ed39-40e5-9350-c47bd51fd6c0' 2021-02-18 10:06:33,390+01 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-81) [test_incremental_backup] Exception: java.lang.ClassCastException: class org.ovirt.engine.core.bll.RedefineVmCheckpointCommand cannot be cast to class org.ovirt.engine.core.bll.SerialChildExecutingCommand (org.ovirt.engine.core.bll.RedefineVmCheckpointCommand and org.ovirt.engine.core.bll.SerialChildExecutingCommand are in unnamed module of loader 'deployment.engine.ear.bll.jar' @176ed0f7) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:29) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) ======================================================================== Any idea? I also now see that basic-suite fails [3], but on a different test - test_import_floating_disk. Not sure that's related. Thanks and best regards, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15615… [2] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15615… [3] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > > > -- > To view, visit https://gerrit.ovirt.org/c/ovirt-system-tests/+/113452 > To unsubscribe, or for help writing mail filters, visit https://gerrit.ovirt.org/settings > > Gerrit-Project: ovirt-system-tests > Gerrit-Branch: master > Gerrit-Change-Id: Ied836cda6b622dbebdb869e0b83fa4c0e0b7ca2c > Gerrit-Change-Number: 113452 > Gerrit-PatchSet: 29 > Gerrit-Owner: Yedidyah Bar David <didi(a)redhat.com> > Gerrit-Reviewer: Anton Marchukov <amarchuk(a)redhat.com> > Gerrit-Reviewer: Dafna Ron <dron(a)redhat.com> > Gerrit-Reviewer: Dusan Fodor <dfodor(a)redhat.com> > Gerrit-Reviewer: Gal Ben Haim <galbh2(a)gmail.com> > Gerrit-Reviewer: Galit Rosenthal <grosenth(a)redhat.com> > Gerrit-Reviewer: Jenkins CI <jenkins(a)ovirt.org> > Gerrit-Reviewer: Name of user not set #1001916 > Gerrit-Reviewer: Yedidyah Bar David <didi(a)redhat.com> > Gerrit-Reviewer: Zuul CI <zuul(a)ovirt.org> > Gerrit-Comment-Date: Thu, 18 Feb 2021 09:17:18 +0000 > Gerrit-HasComments: No > Gerrit-Has-Labels: Yes > Gerrit-MessageType: comment > -- Didi

2 1

ovirt-engine has been tagged (ovirt-engine-4.4.5.6)
by Tal Nisan 17 Feb '21

17 Feb '21

2 1

Support for SSH keys other than RSA
by Artur Socha 12 Feb '21

12 Feb '21

Hi, I have been recently working on adding support for SSH keys other than RSA (communication between ovirt-engine and hosts(VDS-es)). The entire effort is tracked in Bugzilla [1]. There are couple important changes I would like to share with you. First and the most important is changing the way connection is verified. Previously fingerprints (by default SHA-256 unless changed via configuration) were used to verify if the connection between the engine and the host could be established. Now public keys are compared instead (with one exception for backward compatibility). For backward compatibility ie. for previously added (legacy) hosts with fingerprint calculated out of RSA public key (the key not stored in db) the verification is done as before that means we compare fingerprints only. After upgrade the whole setup is expected to work without any manual intervention. However, there are couple of options to 'migrate' legacy fingerprint to whatever ssh server finds the strongest on the host: 1) In database remove sshkeyfingerprint value ie. update vds_static set sshkeyfingerprint='' where vds_id = 'PUT_HERE_HOST_ID' 2) REST:prepare request with blank fingerprint for 'legacy' hosts Please see the (documentation [2]). Fingerprint and public key will be re-entered, 3) reinstall host / install new host 4) manually deploy key and update host's VDS_static.sshkeyfingerprint and vds_static.public_key On engine's UI side there is still a way to fetch fingerprints (on 'New Host' panel but we anticipate that soon there will be a public key (open ssh format) instead. Please let me know if you have any questions, doubts or if you encounter any issues around this area. Patches (referenced in BZ[1]) has been merged into master and this feature is expected to go with 4.4.5 upstream release. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1837221 [2] https://jenkins.ovirt.org/job/ovirt-engine-api-model_standard-check-patch/1… best, Artur

2 1

ansible: Unsupported parameters found in auth
by Yedidyah Bar David 12 Feb '21

12 Feb '21

Hi all, he-basic-suite fails [1] with: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Ensure that the target datacenter is present] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unsupported parameters for (ovirt_datacenter) module: compress, timeout found in auth. Supported parameters include: ca_file, headers, hostname, insecure, kerberos, password, token, url, username"} Any clue? Thanks, [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1918/ -- Didi

2 1

ovirt-engine has been tagged (ovirt-engine-4.4.5.5)
by Tal Nisan 10 Feb '21

10 Feb '21

2 1

reboot after installation during hosted-engine deploy (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 1915 - Failure!)
by Yedidyah Bar David 10 Feb '21

10 Feb '21

Hi all, On Wed, Feb 10, 2021 at 4:34 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1915/ https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/1915… : 2021-02-10 03:21:31,081+0100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:111 TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up] 2021-02-10 03:23:57,041+0100 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': SIG1 Most likely this is because by default we now reboot the host after installation. We need to not do this for the host we deploy on, something like: https://github.com/oVirt/ovirt-ansible-collection/pull/218 Do we have a bug for this? Best regards, > Build Number: 1915 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #1915 > [Yedidyah Bar David] test_utils/host_status_utils: Make dc_name parametric > > [Galit Rosenthal] Add node upgrade suite > > [Ehud Yonasi] run_code: add gpg keys only for el8. > > [Ehud Yonasi] run_code: fixed check for el8. > > [Galit Rosenthal] Move ansible nightly to run on el8 > > > > > ----------------- > Failed Tests: > ----------------- > No tests ran. -- Didi

1 1

"Too many open files" in vdsm.log after 380 migrations
by Yedidyah Bar David 09 Feb '21

09 Feb '21

Hi all, I ran a loop of [1] (from [2]). The loop succeeded for ~ 380 iterations, then failed with 'Too many open files'. First failure was: 2021-02-08 02:21:15,702+0100 ERROR (jsonrpc/4) [storage.HSM] Could not connect to storageServer (hsm:2446) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2443, in connectStorageServer conObj.connect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 449, in connect return self._mountCon.connect() File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 171, in connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File "/usr/lib/python3.6/site-packages/vdsm/storage/mount.py", line 210, in mount cgroup=cgroup) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 56, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda> **kwargs) File "<string>", line 2, in mount File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) OSError: [Errno 24] Too many open files But obviously, once it did, it continued failing for this reason on many later operations. Is this considered a bug? Do we actively try to prevent such cases? So should I open one and attach logs? Or it can be considered a "corner case"? Using vdsm-4.40.50.3-37.git7883b3b43.el8.x86_64 from ost-images-el8-he-installed-1-202102021144.x86_64 . I can also let access to the machine(s) if needed, for now. Thanks and best regards, [1] https://gerrit.ovirt.org/gitweb?p=ovirt-system-tests.git;a=blob;f=he-basic-… [2] https://gerrit.ovirt.org/c/ovirt-system-tests/+/113300 -- Didi

3 8

Network error during communication with the Host (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 857 - Still Failing!)
by Yedidyah Bar David 09 Feb '21

09 Feb '21

On Tue, Feb 9, 2021 at 7:24 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… > Build Number: 857 > Build Status: Still Failing > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #855 > [Eitan Raviv] network: remove fqdn from /etc/hosts > > > Changes for Build #856 > [Eitan Raviv] network: remove fqdn from /etc/hosts > > > Changes for Build #857 > [Eitan Raviv] network: remove fqdn from /etc/hosts > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: basic-suite-master.test-scenarios.test_002_bootstrap.test_add_nfs_master_storage_domain > > Error Message: > AssertionError: False != True after 600 seconds > > Stack Trace: > engine = <ovirtsdk4.services.SystemService object at 0x7fda58b32828> > event_id = [956] > > @contextlib.contextmanager > def wait_for_event(engine, event_id): > ''' > event_id could either be an int - a single > event ID or a list - multiple event IDs > that all will be checked > ''' > events = engine.events_service() > last_event = int(events.list(max=2)[0].id) > try: > > yield > > ../ost_utils/ost_utils/engine_utils.py:38: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > api = <ovirtsdk4.Connection object at 0x7fda58b32748> > domain = <ovirtsdk4.types.StorageDomain object at 0x7fda58d1fac8> > dc_name = 'test-dc' > > def add(api, domain, dc_name): > system_service = api.system_service() > sds_service = system_service.storage_domains_service() > with engine_utils.wait_for_event(system_service, 956): # USER_ADD_STORAGE_DOMAIN(956) > > sd = sds_service.add(domain) > > ../ost_utils/ost_utils/storage_utils/domain.py:33: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.services.StorageDomainsService object at 0x7fda57ca4f60> > storage_domain = <ovirtsdk4.types.StorageDomain object at 0x7fda58d1fac8> > headers = None, query = {}, wait = True, kwargs = {} > > def add( > self, > storage_domain, > headers=None, > query=None, > wait=True, > **kwargs > ): > """ > Adds a new storage domain. > Creation of a new <<types/storage_domain,StorageDomain>> requires the `name`, `type`, `host`, and `storage` > attributes. Identify the `host` attribute with the `id` or `name` attributes. In {product-name} 3.6 and > later you can enable the wipe after delete option by default on the storage domain. To configure this, specify > `wipe_after_delete` in the POST request. This option can be edited after the domain is created, but doing so will > not change the wipe after delete property of disks that already exist. > To add a new storage domain with specified `name`, `type`, `storage.type`, `storage.address`, and `storage.path`, > and using a host with an id `123`, send a request like this: > [source] > ---- > POST /ovirt-engine/api/storageDomains > ---- > With a request body like this: > [source,xml] > ---- > <storage_domain> > <name>mydata</name> > <type>data</type> > <storage> > <type>nfs</type> > <address>mynfs.example.com</address> > <path>/exports/mydata</path> > </storage> > <host> > <name>myhost</name> > </host> > </storage_domain> > ---- > To create a new NFS ISO storage domain send a request like this: > [source,xml] > ---- > <storage_domain> > <name>myisos</name> > <type>iso</type> > <storage> > <type>nfs</type> > <address>mynfs.example.com</address> > <path>/export/myisos</path> > </storage> > <host> > <name>myhost</name> > </host> > </storage_domain> > ---- > To create a new iSCSI storage domain send a request like this: > [source,xml] > ---- > <storage_domain> > <name>myiscsi</name> > <type>data</type> > <storage> > <type>iscsi</type> > <logical_units> > <logical_unit id="3600144f09dbd050000004eedbd340001"/> > <logical_unit id="3600144f09dbd050000004eedbd340002"/> > </logical_units> > </storage> > <host> > <name>myhost</name> > </host> > </storage_domain> > ---- > > > This method supports the following parameters: > > `storage_domain`:: The storage domain to add. > > `headers`:: Additional HTTP headers. > > `query`:: Additional URL query parameters. > > `wait`:: If `True` wait for the response. > """ > # Check the types of the parameters: > Service._check_types([ > ('storage_domain', storage_domain, types.StorageDomain), > ]) > > # Build the URL: > query = query or {} > > # Send the request and wait for the response: > > return self._internal_add(storage_domain, headers, query, wait) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/services.py:26182: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.services.StorageDomainsService object at 0x7fda57ca4f60> > object = <ovirtsdk4.types.StorageDomain object at 0x7fda58d1fac8>, headers = {} > query = {}, wait = True > > def _internal_add(self, object, headers=None, query=None, wait=None): > """ > Executes an `add` method. > """ > # Populate the headers: > headers = headers or {} > > # Send the request and wait for the response: > request = http.Request(method='POST', path=self._path, query=query, headers=headers) > request.body = writer.Writer.write(object, indent=True) > context = self._connection.send(request) > > def callback(response): > if response.code in [200, 201, 202]: > return self._internal_read_body(response) > else: > self._check_fault(response) > > future = Future(self._connection, context, callback) > > return future.wait() if wait else future > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:232: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.service.Future object at 0x7fda57ca49b0> > > def wait(self): > """ > Waits till the result of the operation that created this future is > available. > """ > response = self._connection.wait(self._context) > > return self._code(response) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:55: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > response = <ovirtsdk4.http.Response object at 0x7fda57ca4f98> > > def callback(response): > if response.code in [200, 201, 202]: > return self._internal_read_body(response) > else: > > self._check_fault(response) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:229: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > self = <ovirtsdk4.services.StorageDomainsService object at 0x7fda57ca4f60> > response = <ovirtsdk4.http.Response object at 0x7fda57ca4f98> > > def _check_fault(self, response): > """ > Reads the response body assuming that it contains a fault message, > converts it to an exception and raises it. > > This method is intended for internal use by other > components of the SDK. Refrain from using it directly, > as backwards compatibility isn't guaranteed. > """ > > body = self._internal_read_body(response) > if isinstance(body, types.Fault): > > self._raise_error(response, body) > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:132: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > response = <ovirtsdk4.http.Response object at 0x7fda57ca4f98> > detail = <ovirtsdk4.types.Fault object at 0x7fda57ca44e0> > > @staticmethod > def _raise_error(response, detail=None): > """ > Creates and raises an error containing the details of the given HTTP > response and fault. > > This method is intended for internal use by other components of the > SDK. Refrain from using it directly, as backwards compatibility isn't > guaranteed. > """ > fault = detail if isinstance(detail, types.Fault) else None > > msg = '' > if fault: > if fault.reason: > if msg: > msg += ' ' > msg = msg + 'Fault reason is "%s".' % fault.reason > if fault.detail: > if msg: > msg += ' ' > msg = msg + 'Fault detail is "%s".' % fault.detail > if response: > if response.code: > if msg: > msg += ' ' > msg = msg + 'HTTP response code is %s.' % response.code > if response.message: > if msg: > msg += ' ' > msg = msg + 'HTTP response message is "%s".' % response.message > > if isinstance(detail, six.string_types): > if msg: > msg += ' ' > msg = msg + detail + '.' > > class_ = Error > if response is not None: > if response.code in [401, 403]: > class_ = AuthError > elif response.code == 404: > class_ = NotFoundError > > error = class_(msg) > error.code = response.code if response else None > error.fault = fault > > raise error > E ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Did anyone check this? It started failing yesterday. Also for me, e.g.: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15458/ Thanks and best regards, > > /usr/lib64/python3.6/site-packages/ovirtsdk4/service.py:118: Error > > During handling of the above exception, another exception occurred: > > engine_api = <ovirtsdk4.Connection object at 0x7fda58b32748> > sd_nfs_host_storage_ip = '192.168.202.2' > > @order_by(_TEST_LIST) > @pytest.mark.skipif(MASTER_SD_TYPE != 'nfs', reason='not using nfs') > def test_add_nfs_master_storage_domain(engine_api, sd_nfs_host_storage_ip): > > add_nfs_storage_domain(engine_api, sd_nfs_host_storage_ip) > > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:545: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ../basic-suite-master/test-scenarios/test_002_bootstrap.py:554: in add_nfs_storage_domain > nfs_version='v4_2') > ../ost_utils/ost_utils/storage_utils/nfs.py:62: in add_domain > domain.add(engine_api, p, dc_name) > ../ost_utils/ost_utils/storage_utils/domain.py:37: in add > lambda: sd_service.get().status == sdk4.types.StorageDomainStatus.UNATTACHED > /usr/lib64/python3.6/contextlib.py:99: in __exit__ > self.gen.throw(type, value, traceback) > ../ost_utils/ost_utils/engine_utils.py:44: in wait_for_event > lambda: > ../ost_utils/ost_utils/assertions.py:98: in assert_true_within_long > assert_equals_within_long(func, True, allowed_exceptions) > ../ost_utils/ost_utils/assertions.py:83: in assert_equals_within_long > func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > func = <function wait_for_event.<locals>.<lambda> at 0x7fda57c9c2f0> > value = True, timeout = 600, allowed_exceptions = [], initial_wait = 0 > error_message = 'False != True after 600 seconds' > > def assert_equals_within( > func, value, timeout, allowed_exceptions=None, initial_wait=10, > error_message=None > ): > allowed_exceptions = allowed_exceptions or [] > res = '<no-result-obtained>' > with _EggTimer(timeout) as timer: > while not timer.elapsed(): > try: > res = func() > if res == value: > return > except Exception as exc: > if _instance_of_any(exc, allowed_exceptions): > time.sleep(3) > continue > > LOGGER.exception("Unhandled exception in %s", func) > raise > > if initial_wait == 0: > time.sleep(3) > else: > time.sleep(initial_wait) > initial_wait = 0 > try: > if error_message is None: > error_message = '%s != %s after %s seconds' % (res, value, timeout) > > raise AssertionError(error_message) > E AssertionError: False != True after 600 seconds > > ../ost_utils/ost_utils/assertions.py:61: AssertionError_______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/ND5KFKIA7BNG7… -- Didi

1 0

Connection issues during send request
by Yedidyah Bar David 08 Feb '21

08 Feb '21

Hi all, Please see [1]. engine.log has many errors like: 2021-02-08 14:01:57,529+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (EE-ManagedThreadFactory-engine-Thread-2) [c14cce8] Error: org.ovirt.engine.core.vdsbroker.TransportRunTimeException: Connection issues during send request vdsm.log seems ok. Didn't try to investigate much, just tried 'ci test' again on the patch, for now. Any idea? [1] https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/15443/ -- Didi

2 1

Issue running OST basic suite master
by Galit Rosenthal 04 Feb '21

04 Feb '21

Hi, Basic suite master is failing [1] on basic-suite-master.test-scenarios.test_004_basic_sanity.test_import_vm1 Can someone have a look? Regards, Galit [1] https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_el8nigh… -- Galit Rosenthal SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit(a)redhat.com T: 972-9-7692230 <https://www.redhat.com/>

3 2

ovirt-engine has been tagged (ovirt-engine-4.4.5.4)
by Tal Nisan 03 Feb '21

03 Feb '21

1 0

Switch CentOS to Oracle Linux
by ntoxicator＠gmail.com 02 Feb '21

02 Feb '21

Hello, Oracle Linux is offering CentOS users/dev/production servers to switch to Oracle Linux free of cost. They've produced a script to do this. Oracle also has their own Repo's for oVirt and use it for their own Oracle Linux Virtualization Manager [Re-branded oVirt]. I know Red Hat has announced moving away from RHV and focusing on OpenShift virtualization. They'll happen in 2022-forward. Are we already to assume oVirt will follow suite and migrate to OpenShift?? Questions: Since announcement of CentOS going EOL and move to CentOS Stream... [Not doing it]. - Will the oVirt team continue to develop past 4.4 release as well as support Oracle Linux? - Does Oracle really develop to their own ovirt repo and virtualization manager product - What about supporting OpenSuSE or SLES? - oVirt packages on RHEL? [RH announced they will allow RHEL installed on up to 16-hosts for FREE] I am considering to test moving CentOS 7 development work-loads to Oracle Linux and use their oVirt repo. Really like to know if oVirt developers plan to continue to build on what is there moving forward. Or will the project die? I use oVirt on production work-loads [same as others] with GREAT success. This is an absolute VMWare replacement.

3 3

[CI] Wrong version of qemu-kvm in the CI on ppc64le
by Nir Soffer 02 Feb '21

02 Feb '21

We have tests that used to pass 2-3 weeks ago, but started to fail today on ppc64le with this error: qemu-kvm: Requested safe cache capability level not supported by kvm, try appending -machine cap-cfpc=broken Full log: https://jenkins.ovirt.org/job/ovirt-imageio_standard-check-patch/3599//arti… I can add -machine cap-cfpc=broken, but it looks like we are running the wrong qemu version in CI: 17:31:08,513 DEBUG (MainThread) [qmp] Talking with qemu 4.2.0 We should use 5.1.0, available on Centos 8.3.2011: $ dnf info qemu-kvm ... Available Packages Name : qemu-kvm Epoch : 15 Version : 5.1.0 Release : 14.el8.1 Architecture : x86_64 Size : 126 k Source : qemu-kvm-5.1.0-14.el8.1.src.rpm Repository : ovirt-master-advanced-virtualization-testing Is it possible that we don't have a build for ppc6le, or something is broken in this build? Since the tests used to pass, I don't think this is a repo issue. Nir

1 2

CentOS Stream support
by Michal Skrivanek 01 Feb '21

01 Feb '21

Hi all, we would like to ask about interest in community about oVirt moving to CentOS Stream. There were some requests before but it’s hard to see how many people would really like to see that. With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS. So..would you like us to support CentOS Stream? We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS? There are some concerns about Stream being a bit less stable, do you share those concerns? Thank you for your comments, michal

6 12