February 2024 - Users - oVirt List Archives

IO per VM monitoring
by marek 19 Feb '24

19 Feb '24

hi, i have prometheus based ovirt hosts monitoring (node_exporter, smartcl_exporter, ipmi_exporter) https://prometheus-community.github.io/ansible/branch/main/ and alerts from https://samber.github.io/awesome-prometheus-alerts/ after i started this monitoring i found that one VM is overloading local storage (so i must check IO limiting documentation as a homework :) ) but my question is how do you monitor IO traffic per VM? (IOPS, read/write traffic,..) some qemu/libvirt exporter? some custom text file + node_exporter? thanks for tips Marek

4 7

Ovirt 4.4.10 - Admin account locked after password change
by Andy Lau 19 Feb '24

19 Feb '24

Hello All Recently, I reset the password for the admin account through ovirt-aaa-jdbc-tool. The password changed properly but the admin account will be locked automatically after few hours later. From the engine log, the admin@internal seems login periodically from ovirt engine itself. If I fallback the password with the original one, all working fine. Do anyone has the solution? Screen capture from the engine log: https://polyuit-my.sharepoint.com/:i:/g/personal/tflau_polyu_edu_hk/Edqp2ls…

3 4

Urgent: Assistance Needed - oVirt Host Unresponsive
by Sachendra Shukla 18 Feb '24

18 Feb '24

HI Team, I am writing to inform you about an issue we are currently facing with our oVirt host. Unfortunately, the host has become unresponsive, and our attempts to place it in maintenance mode have been unsuccessful. Additionally, when checking the VDSM status, we encountered an error. Here is a snapshot for your reference: [image: image.png] VDSM error - [image: image.png] -- Regards, Sachendra Shukla Yagna iQ, Inc. and subsidiaries HQ Address: Yagna iQ Inc. 7700 Windrose Ave, Suite G300, Plano, TX 75024, USA 75024, Website: https://yagnaiq.com Contact Customer Support: support(a)yagnaiq.com Privacy Policy: https://www.yagnaiq.com/privacy-policy/ *This communication and any attachments may contain confidential information and/or copyright material of Yagna iQ, Inc. * All unauthorized use, disclosure or distribution is prohibited. If you are not the intended recipient, please notify Yagna iQ immediately by replying to the email and destroy all copies of this communication. This email has been scanned for all known viruses. The sender does not accept liability for any damage inflicted by viewing the content of this email.

5 7

change iscsi auth on hosted engine storage domain
by Jirka Simon 16 Feb '24

16 Feb '24

Hello oVirt community, I have hosted engine on iSCSI storage without any authentication, I would like to change it. Is there any way how to do it without redeploy hosted engine ? Thank you for any help. Jirka

1 0

mirrorlist.ovirt.org Repository is not working
by raghawendra.m＠gmail.com 14 Feb '24

14 Feb '24

Hi, This morning(14Feb2024) noticed that, ovirt repositories are not accessible. Status code: 503 for https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el9 (IP: 8.43.85.224) Hopefully, you are in process of fixing that issue. Regards, Raghavendra M

1 0

Internal pentest result : Ovirt-engine authentication bypass
by Jirka Simon 13 Feb '24

13 Feb '24

Hello ovirt comunity. We had an internal pentest here and one finding is *Ovirt-engine authentication bypass.* Ovirt-engine, as deployed on ovirtm.XXX.XXX.cz, contains an authentication bypass. It is possible to directly call the CreateUserSessionCommand using runAction exposed by /ovirt- engine/webadmin/GenericApiGWTService. *This action explicitly enables everyone to call it:* ```/ @Override protected boolean isUserAuthorizedToRunAction() { return true; } /``` The behavior of this call differs based on the ENGINE_SSO_ENABLE_EXTERNAL_SSO configuration option: ``` /boolean externalSsoEnabled = EngineLocalConfig.getInstance().getBoolean("ENGINE_SSO_ENABLE_EXTERNAL_SSO"); DbUser dbUser = externalSsoEnabled ? dbUserDao.getByUsernameAndDomain(params.getPrincipalName(), authzName) : dbUserDao.getByExternalId(authzName, params.getPrincipalId());/ ``` If this option is enabled, usernames are used to locate users. If it's disabled, the externalId (which seems to be a randomly generated GUID) is used to locate users. If the specified user exists, a session is returned for the user. If the specified user doesn't exist, the user is created in the system. However, the user doesn't get assigned any group membership or rights, therefore the session creation fails because of the missing Login right. The attempt to modify the users table can be seen in the SQL error message when attempting to use a null value for the username (as the endpoint uses GWT, the payload is mostly unreadable): ``` /POST /ovirt-engine/webadmin/GenericApiGWTService HTTP/1.1 Host: ovirtm.xxx.xxx.cz 14 Final Report: Results of penetration testing (internal, external, Wi-Fi) 21 December 2023 Cookie: JSESSIONID=wsp3WAo63LZGHfpB__stEt4lZ7z_zZycpzIprNlT.ovirtm45; Content-Type: text/x-gwt-rpc; charset=utf-8 X-GWT-Module-Base: https://ovirtm.xxx.xx.cz/ovirt-engine/webadmin X-GWT-Permutation: D7ECB5EF5E29205D18271CC08183A28D Ovirt-Xsrf-Token: 4D87D03B631F8506FC668AA4C3FE3F443D723A9F379FDBB8B0D6DA0668650375 Content-Length: 869 7|0|23|https://ovirtm.xxx.xxx.cz/ovirt- engine/webadmin|0D1B4DEE9D1424E18C443F1CD1C11574|org.ovirt.engine.ui.frontend.gwtservices.GenericApiGWT Service|runAction|org.ovirt.engine.core.common.action.ActionType/2930387551|org.ovirt.engine.core.commo n.action.ActionParametersBase/2903049429|org.ovirt.engine.core.common.action.CreateUserSessionParameter s/2744166832|appScope|email|firstName|java.util.ArrayList/4159755760|lastName|namespace|principalId|adm in|internal|sourceIp|ssoScope|ssoToken|org.ovirt.engine.core.common.action.ActionParametersBase$EndProc edure/1568822488|java.util.Collections$EmptyMap/4174664486|org.ovirt.engine.core.common.businessentitie s.VDSStatus/1938301532|org.ovirt.engine.core.compat.TransactionScopeOption/1475850853|1|2|3|4|2|5|6|5|2 01|7|0|8|9|10|11|0|12|13|14|0|16|17|18|19|0|5|0|0|0|0|20|1|0|11|0|0|0|0|0|0|21|0|- 4|22|0|1|0|1|23|2|0|0|0| HTTP/1.1 200 OK Date: Fri, 15 Dec 2023 09:42:35 GMT Server: Apache/2.4.37 (CentOS Stream) OpenSSL/1.1.1k mod_auth_gssapi/1.6.1 Expires: Thu, 14 Dec 2023 09:42:35 GMT Cache-Control: no-cache, no-store, must-revalidate Set-Cookie: locale=cs_CZ; path=/; secure; HttpOnly; Max-Age=2147483647; Expires=Wed, 02-Jan-2092 12:56:42 GMT X-XSS-PROTECTION: 1; MODE=BLOCK Pragma: no-cache X-FRAME-OPTIONS: SAMEORIGIN Content-Disposition: attachment X-CONTENT-TYPE-OPTIONS: NOSNIFF Content-Length: 1794 Content-Type: application/json;charset=utf-8 Correlation-Id: 664c1c1f-9a75-4e14-94d7-aba12c5442f5 Connection: close //OK[0,5,4,8,3,1,2,474,7,6,1,0,2,0,2,5,1,0,4,3,1,2,0,2,1,1,["org.ovirt.engine.core.common.action.Action ReturnValue/4163585948","java.util.ArrayList/4159755760","java.lang.String/2004016611","ENGINE","","org .ovirt.engine.core.common.errors.EngineFault/2377218566","org.ovirt.engine.core.common.errors.EngineErr or/2640515959","ERROR: null value in column \"username\" violates not-null constraint\n Detail: Failing row contains (6dad5e2f-7c95-4547-8f08-6936494c91b6, firstName, lastName, internal-authz, null, , email, , f, principalId, 2023-12-14 17:51:04.757747+01, 2023-12-15 10:42:35.125994+01, namespace, firstName(a)internal-authz).\n Where: SQL statement \"UPDATE users\n SET department \u003D v_department,\n domain \u003D v_domain,\n email \u003D v_email,\n name \u003D v_name,\n note \u003D v_note,\n surname \u003D v_surname,\n username \u003D v_username,\n external_id \u003D v_external_id,\n namespace \u003D v_namespace,\n _update_date \u003D CURRENT_TIMESTAMP\n WHERE external_id \u003D v_external_id\n AND domain \u003D v_domain\"\nPL/pgSQL function updateuserimpl(character varying,character varying,character varying,character varying,character varying,character varying,uuid,character varying,text,character varying) line 5 at SQL statement\nSQL statement \"SELECT UpdateUserImpl(\n v_department,\n v_domain,\n v_email,\n v_name,\n v_note,\n v_surname,\n v_user_id,\n v_username,\n v_external_id,\n v_namespace)\"\nPL/pgSQL function updateuser(character varying,character varying,character varying,character varying,character varying,character varying,uuid,character varying,boolean,text,character varying) line 3 at PERFORM"],0,7]/ ``` Fortunately, in our deplyoment the ENGINE_SSO_ENABLE_EXTERNAL_SSO configuration was set to false, so to create a session for the admin it would be necessary to know the admin's user externalId. However, as this is not the default configuration, it is possible that a later reinstallation could change the value. Still, it was possible to create users in the system without any authentication. What is the best way to report this security issue? Thank you Jirka

2 3

hosted-engine --deploy fails, apparently due to not finding storage pool
by iucounu＠gmail.com 09 Feb '24

09 Feb '24

Hi, I'm trying to deploy a self hosted engine, and it gets to 'Wait for the host to be up' and then fails about 20mins later. Checking the hosted engine setup log, the failure seems to be to do with a storage pool not being found (see error below). Is this error referring to the storage pool that the hosted engine will use? The setup script does not ask me what kind of storage I want to use, or the path for it, does it auto detect this? The host I am running the engine setup on is part of a gluster cluster, which I have provisioned for this purpose. Do I need to edit a file somewhere to specify this, or is there something else I need to do (permissions etc)? The hosted engine VM itself comes up fine, and I can get a console on it via virsh. Thanks for any help, kind regards, Cam 2024-02-08 16:39:04,438+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'changed': True, 'stdout': '', 'stderr': "error: failed to get pool 'localvm0bod0wzw'\nerror: Storage pool not found: no storage pool with matching name 'localvm0bod0wzw'", 'rc': 1, 'cmd': ['virsh', '-c', 'qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf', 'pool-destroy', 'localvm0bod0wzw'], 'start': '2024-02-08 16:39:04.246510', 'end': '2024-02-08 16:39:04.301790', 'delta': '0:00:00.055280', 'msg': 'non-zero return code', 'invocation': {'module_args': {'_raw_params': 'virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf pool-destroy localvm0bod0wzw', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ["error: failed to get pool 'localvm0bod0wzw'", "error: Storage pool not fou nd: no storage pool with matching name 'localvm0bod0wzw'"], '_ansible_no_log': None} 2024-02-08 16:39:04,539+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localhost]: FAILED! => {"changed": true, "cmd": ["virsh", "-c", "qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf", "pool-destroy", "localvm0bod0wzw"], "delta": "0:00:00.055280", "end": "2024-02-08 16:39:04.301790", "msg": "non-zero return code", "rc": 1, "start": "2024-02-08 16:39:04.246510", "stderr": "error: failed to get pool 'localvm0bod0wzw'\nerror: Storage pool not found: no storage pool with matching name 'localvm0bod0wzw'", "stderr_lines": ["error: failed to get pool 'localvm0bod0wzw'", "error: Storage pool not found: no storage pool with matching name 'localvm0bod0wzw'"], "stdout": "", "stdout_lines": []} 2024-02-08 16:39:04,839+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'changed': True, 'stdout': '', 'stderr': "error: failed to get pool 'localvm0bod0wzw'\nerror: Storage pool not found: no storage pool with matching name 'localvm0bod0wzw'", 'rc': 1, 'cmd': ['virsh', '-c', 'qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf', 'pool-undefine', 'localvm0bod0wzw'], 'start': '2024-02-08 16:39:04.638774', 'end': '2024-02-08 16:39:04.691953', 'delta': '0:00:00.053179', 'msg': 'non-zero return code', 'invocation': {'module_args': {'_raw_params': 'virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf pool-undefine localvm0bod0wzw', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ["error: failed to get pool 'localvm0bod0wzw'", "error: Storage pool not f ound: no storage pool with matching name 'localvm0bod0wzw'"], '_ansible_no_log': None} 2024-02-08 16:39:04,940+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localhost]: FAILED! => {"changed": true, "cmd": ["virsh", "-c", "qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf", "pool-undefine", "localvm0bod0wzw"], "delta": "0:00:00.053179", "end": "2024-02-08 16:39:04.691953", "msg": "non-zero return code", "rc": 1, "start": "2024-02-08 16:39:04.638774", "stderr": "error: failed to get pool 'localvm0bod0wzw'\nerror: Storage pool not found: no storage pool with matching name 'localvm0bod0wzw'", "stderr_lines": ["error: failed to get pool 'localvm0bod0wzw'", "error: Storage pool not found: no storage pool with matching name 'localvm0bod0wzw'"], "stdout": "", "stdout_lines": []} 2024-02-08 16:39:05,141+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': 'The task includes an option with an undefined variable. The error was: \'local_vm_disk_path\' is undefined. \'local_vm_disk_path\' is undefined\n\nThe error appears to be in \'/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/clean_local_storage_pools.yml\': line 16, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n changed_when: true\n - name: Destroy local storage-pool {{ local_vm_disk_path.split(\'/\')[5] }}\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - "{{ foo }}"\n', '_ansible_no_log': False} 2024-02-08 16:39:05,241+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'local_vm_disk_path' is undefined. 'local_vm_disk_path' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/clean_local_storage_pools.yml': line 16, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n changed_when: true\n - name: Destroy local storage-pool {{ local_vm_disk_path.split('/')[5] }}\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"} 2024-02-08 16:39:05,442+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': 'The task includes an option with an undefined variable. The error was: \'local_vm_disk_path\' is undefined. \'local_vm_disk_path\' is undefined\n\nThe error appears to be in \'/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/clean_local_storage_pools.yml\': line 22, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n changed_when: true\n - name: Undefine local storage-pool {{ local_vm_disk_path.split(\'/\')[5] }}\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - "{{ foo }}"\n', '_ansible_no_log': False} 2024-02-08 16:39:05,543+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'local_vm_disk_path' is undefined. 'local_vm_disk_path' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/clean_local_storage_pools.yml': line 22, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n changed_when: true\n - name: Undefine local storage-pool {{ local_vm_disk_path.split('/')[5] }}\n ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes. Always quote template expression brackets when they\nstart a value. For instance:\n\n with_items:\n - {{ foo }}\n\nShould be written as:\n\n with_items:\n - \"{{ foo }}\"\n"} 2024-02-08 16:39:05,643+0000 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 PLAY RECAP [localhost] : ok: 22 changed: 4 unreachable: 0 skipped: 2 failed: 0

1 0

Unable to upload images
by avegab＠indra.es 07 Feb '24

07 Feb '24

Hello friends! I can't upload images; when I click on test connection, I encounter the certificate issue. I used to be able to upload images without any problem, but this issue started when I updated the certificates of the Hosts and reinstalled one of the Hosts so it could also deploy the engine. I am sure that the certificate is registered in the browser. Could you help me? Thank you. ovirt-imageio --show-config { "backend_file": { "buffer_size": 8388608 }, "backend_http": { "buffer_size": 8388608, "ca_file": "/etc/pki/ovirt-engine/ca.pem" }, "backend_nbd": { "buffer_size": 8388608 }, "control": { "port": 54324, "prefer_ipv4": true, "remove_timeout": 60, "socket": "/run/ovirt-imageio/sock", "transport": "tcp" }, "daemon": { "drop_privileges": true, "group_name": "ovirtimg", "max_connections": 8, "poll_interval": 1.0, "run_dir": "/run/ovirt-imageio", "user_name": "ovirtimg" }, "formatter_long": { "format": "%(asctime)s %(levelname)-7s (%(threadName)s) [%(name)s] %(message)s" }, "formatters": { "keys": "long" }, "handler_logfile": { "args": "(\"/var/log/ovirt-imageio/daemon.log\",)", "formatter": "long", "class": "logging.handlers.RotatingFileHandler", "kwargs": "{\"maxBytes\": 20971520, \"backupCount\": 10}", "level": "DEBUG" }, "handler_stderr": { "args": "()", "formatter": "long", "class": "logging.StreamHandler", "level": "DEBUG" }, "handlers": { "keys": "logfile" }, "local": { "enable": false, "socket": "\u0000/org/ovirt/imageio" }, "logger_root": { "handlers": "logfile", "level": "INFO", "propagate": 0 }, "loggers": { "keys": "root" }, "profile": { "filename": "/run/ovirt-imageio/profile" }, "remote": { "host": "::", "port": 54323 }, "tls": { "ca_file": "/etc/pki/ovirt-engine/apache-ca.pem", "cert_file": "/etc/pki/ovirt-engine/certs/apache.cer", "enable": true, "enable_tls1_1": false, "key_file": "/etc/pki/ovirt-engine/keys/apache.key.nopass" } }

1 0

Enable hpet on specific vm
by ricardoot＠gmail.com 07 Feb '24

07 Feb '24

Hello friends, Please. Does anyone know how to use hooks to activate hpet in a VM? I have tried using qemu_cmdline with this configuration ["-rtc", "hpet,timer-states=3"] and with this other ["-rtc", "hpet= on"] but it gives me this error: "VM w2022 is down with error. Exit message: internal error: process exited while connecting to monitor: qemu-kvm: -rtc hpet=on: Invalid parameter 'hpet'." And VM not starup. I have read that https://www.ovirt.org/develop/developer-guide/vdsm/hooks.html and try to use: https://www.ovirt.org/develop/developer-guide/vdsm/hook/qemucmdline.html https://novamostra.com/2021/05/11/make-ovirt-vm-reflect-host-hardware-prope… I've been trying to solve this for a long time. I wrote another post that brought me to try using hooks: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/VGUARICFTRVHJA… Thanks so much.

1 0

Problem with ovirt-websocket-proxy updated certified.
by Kalil de A. Carvalho 04 Feb '24

04 Feb '24

Hello all. In my company we have a oVit 4.5.4 that was worrying fine until today when the Apache certified expereid. Looking for how I can updated ti, followed here: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/ZI5WNU6OB6FZMQ… I could access my engine. A secunde problem apeready when me and my colleagues tried to access any VM through noVNC On the web tab show us this message: "Something went wrong, connection is closed" When I went on the engine discovery that Websocket-proxy certified was expired too. I tried tha same procedure above but on the tab was the same message but on /var/log/message show us this: Feb 2 16:30:46 engine journal[5391]: 2024-02-02 16:30:46,072-0300 ovirt-websocket-proxy: INFO msg:601 handler exception: [SSL: TLSV1_ALERT_UNKNOWN_C A] tlsv1 alert unknown ca (_ssl.c:1129) Feb 2 16:30:46 engine ovirt-websocket-proxy.py[5391]: ovirt-websocket-proxy[5391] INFO msg:601 handler exception: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv 1 alert unknown ca (_ssl.c:1129) I updated the CA-CERTIFITES, re-checked all paths to be sure that was correct but nothing worked. Has anyone passed through this problem? Best regards, -- Atenciosamente, Kalil de A. Carvalho

2 1