January 2024 - Users - oVirt List Archives

oVirt nodes with local storage
by Wild Star 21 Jan '24

21 Jan '24

On the weekend I upgraded my self-hosted oVirt engine to 4.56. All went well with that! I also spotted that there was an update for all my 4.54 nodes, but something has changed with the repo overnight because none of my nodes see updates anymore, though the update remains available here … https://resources.ovirt.org/pub/ovirt-4.5/iso/ovirt-node-ng-installer/. Yesterday, I attempted to update just one of my nodes and ran into this snag… “Local storage domains were found on the same filesystem as / ! Please migrate the data to a new LV before upgrading, or you will lose the VMs”. I’ve always stored my VMs in a separate /DATA directory off the root filesystem, and shared the local storage using NFS. I know it’s not ideal, but with good frequent regular backups, it has served me well for many years. A Google search revealed, that others have also had similar local storage issues, and suggestions on ways to mitigate the issue, including the oVirt documentation found here… https://www.ovirt.org/documentation/upgrade_guide/index.html#Upgrading_hype…, is not my preferred fix. In the past node updates with my local storage were not a problem and were easy peasy! From some of the discussions I saw at Red Hat, I have deduced (maybe wrongly) that there was an issue that necessitated a fix, which introduced a required check for local storage during the upgrade process. I probably should move all the local storage off the oVirt nodes, but at this time, that is easier said than done. I’m posting here only to see if there are perhaps other ideas or perspectives I may not have thought of and should consider with my local storage. Thank you all in advance, much appreciated, and of course, thank you all for supporting oVirt!

3 5

Odd behavior after upgrade to 4.5
by Seann G. Clark 19 Jan '24

19 Jan '24

All, I just recently upgraded from 4.3 to 4.5, and while most of the upgrade went smoothly and I didn't see any major issues, I have noticed a few things that are causing problems. The one piece that I had to sneak in to make the install successful, is push my CA and intermediate CA certs into the engine after it came up the first time, but before it fully installed, otherwise it would fail restoring the backup. This is to address using a custom Apache cert for the hosted engine, instead of the default self-signed CA. The biggest issue is with keycloak. To make the upgrade succeed I had to skip keycloak installation, after doing some digging related to this error: "Failed to execute stage 'Misc configuration': 'OVESETUP_OVN/ovirtProviderOvnSecret'" I don't actively use OVN in my environment right now, and I think it is all pretty much in the default install state. This is the same error I get when I loop back to install keycloak now that the engine is up. This is what is in the logs for the last few lines preceding the error and failure event: 2024-01-19 19:09:32,602+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn plugin.execute:921 execute-output: ('ovn-sbctl', 'set-connection', 'pssl:6642:[::]') stdout: 2024-01-19 19:09:32,602+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn plugin.execute:926 execute-output: ('ovn-sbctl', 'set-connection', 'pssl:6642:[::]') stderr: 2024-01-19 19:09:32,626+0000 DEBUG otopi.context context._executeMethod:127 Stage misc METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_ovn_timeout 2024-01-19 19:09:32,627+0000 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn ovirtproviderovn._misc_configure_ovn_timeout:1076 Updating OVN timeout configuration 2024-01-19 19:09:32,628+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn plugin.executeRaw:813 execute: ('ovn-sbctl', 'set', 'connection', '.', 'inactivity_probe=60000'), executable='None', cwd='None', env=None 2024-01-19 19:09:32,652+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn plugin.executeRaw:863 execute-result: ('ovn-sbctl', 'set', 'connection', '.', 'inactivity_probe=60000'), rc=0 2024-01-19 19:09:32,653+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn plugin.execute:921 execute-output: ('ovn-sbctl', 'set', 'connection', '.', 'inactivity_probe=60000') stdout: 2024-01-19 19:09:32,653+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn plugin.execute:926 execute-output: ('ovn-sbctl', 'set', 'connection', '.', 'inactivity_probe=60000') stderr: 2024-01-19 19:09:32,677+0000 DEBUG otopi.context context._executeMethod:127 Stage misc METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.network.ovirtproviderovn.Plugin._misc_configure_provider 2024-01-19 19:09:32,678+0000 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/network/ovirtproviderovn.py", line 1113, in _misc_configure_provider self._configure_ovirt_provider_ovn() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/network/ovirtproviderovn.py", line 796, in _configure_ovirt_provider_ovn content = self._create_config_content() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/network/ovirtproviderovn.py", line 761, in _create_config_content OvnEnv.OVIRT_PROVIDER_OVN_SECRET KeyError: 'OVESETUP_OVN/ovirtProviderOvnSecret' 2024-01-19 19:09:32,682+0000 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc configuration': 'OVESETUP_OVN/ovirtProviderOvnSecret' 2024-01-19 19:09:32,683+0000 DEBUG otopi.transaction transaction.abort:124 aborting 'DNF Transaction' I have spent a fair amount of time researching this but haven't found a specific root cause, or solution on this issue. Thank you in advance, Seann

1 0

Cannot update oVirt Node
by Gianluca Amato 19 Jan '24

19 Jan '24

Hello, I am trying to update my oVirt installation to 4.5.5, but while I had no problems upgrading the self-hosted ovirt-engine, I am not able to upgrade any node (running oVirt Node NG 4.5.4 or older). I just click Upgrade in the oVirt Manager, and I get a bunch of events telling me that everything was OK and the upgrade succeeded, but in reality nothing happened on the nodes: the result of "nodectl info" before and after the upgrade process is exactly the same. I gave a look to the engine.log and to the relevant ovirt-host-mgmt-ansible log in the ovirt-engine VM, but I cannot find the problem. Do any of you have any ideas ? I am attaching some relevant files. Note that the upgrade consistently fails in the same way on other nodes, which have other versions of oVirt Node NG (always in the 4.5 family). Thanks for your help --gianluca

5 7

hosted-engine --deploy: No module named 'he_ansible'
by iucounu＠gmail.com 19 Jan '24

19 Jan '24

Hi, I'm trying to install hosted engine on a server (4.5.5-1.el9) and it reports 'No module named 'he_ansible'. I have what could an he_ansible module under /usr/share/ovirt-hosted-engine-setup: /usr/share/ovirt-hosted-engine-setup/he_ansible /usr/share/ovirt-hosted-engine-setup/he_ansible/__pycache__ /usr/share/ovirt-hosted-engine-setup/he_ansible/ansible.cfg /usr/share/ovirt-hosted-engine-setup/he_ansible/callback_plugins /usr/share/ovirt-hosted-engine-setup/he_ansible/constants.py /usr/share/ovirt-hosted-engine-setup/he_ansible/trigger_role.yml /usr/share/ovirt-hosted-engine-setup/he_ansible/__pycache__/constants.cpython-39.opt-1.pyc /usr/share/ovirt-hosted-engine-setup/he_ansible/__pycache__/constants.cpython-39.pyc /usr/share/ovirt-hosted-engine-setup/he_ansible/callback_plugins/1_otopi_json.py /usr/share/ovirt-hosted-engine-setup/he_ansible/callback_plugins/2_ovirt_logger.py Is there a missing include/path somewhere? Thanks for any help. Full error: bash# hosted-engine --deploy <string>:1: DeprecationWarning: distro.linux_distribution() is deprecated. It should only be used as a compatibility shim with Python's platform.linux_distribution(). Please use distro.id(), distro.version() and distro.name() instead. <string>:1: DeprecationWarning: distro.linux_distribution() is deprecated. It should only be used as a compatibility shim with Python's platform.linux_distribution(). Please use distro.id(), distro.version() and distro.name() instead. ***L:ERROR Internal error: No module named 'he_ansible' Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/otopi/main.py", line 141, in execute self.context.loadPlugins() File "/usr/lib/python3.9/site-packages/otopi/context.py", line 803, in loadPlugins self._loadPluginGroups(plugindir, needgroups, loadedgroups) File "/usr/lib/python3.9/site-packages/otopi/context.py", line 112, in _loadPluginGroups self._loadPlugins(path, path, groupname) File "/usr/lib/python3.9/site-packages/otopi/context.py", line 69, in _loadPlugins self._loadPlugins(base, d, groupname) File "/usr/lib/python3.9/site-packages/otopi/context.py", line 95, in _loadPlugins util.loadModule( File "/usr/lib/python3.9/site-packages/otopi/util.py", line 110, in loadModule spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module> from . import misc File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 34, in <module> from ovirt_hosted_engine_setup import ansible_utils File "/usr/lib/python3.9/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 33, in <module> from he_ansible.constants import AnsibleCallback ModuleNotFoundError: No module named 'he_ansible' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/otopi/__main__.py", line 88, in main installer.execute() File "/usr/lib/python3.9/site-packages/otopi/main.py", line 143, in execute util.raiseExceptionInformation( File "/usr/lib/python3.9/site-packages/otopi/util.py", line 85, in raiseExceptionInformation raise info[1].with_traceback(info[2]) File "/usr/lib/python3.9/site-packages/otopi/main.py", line 141, in execute self.context.loadPlugins() File "/usr/lib/python3.9/site-packages/otopi/context.py", line 803, in loadPlugins self._loadPluginGroups(plugindir, needgroups, loadedgroups) File "/usr/lib/python3.9/site-packages/otopi/context.py", line 112, in _loadPluginGroups self._loadPlugins(path, path, groupname) File "/usr/lib/python3.9/site-packages/otopi/context.py", line 69, in _loadPlugins self._loadPlugins(base, d, groupname) File "/usr/lib/python3.9/site-packages/otopi/context.py", line 95, in _loadPlugins util.loadModule( File "/usr/lib/python3.9/site-packages/otopi/util.py", line 110, in loadModule spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/__init__.py", line 25, in <module> from . import misc File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 34, in <module> from ovirt_hosted_engine_setup import ansible_utils File "/usr/lib/python3.9/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 33, in <module> from he_ansible.constants import AnsibleCallback otopi.main.PluginLoadException: No module named 'he_ansible' root@lonovirt1 /u/cmcl bash#

1 1

virt-viewer - lost partialy connectivity
by paf1＠email.cz 19 Jan '24

19 Jan '24

Hello, after update centos9-stream to latest packages with ovirt 4.5.5 I lost VMs console connectivity . For html5 option I lost all ( can't connect anyway ) For virt-viewer option only partially, can view output screen, but keyboard has no input ( sometimes accept enter ) I tried to reinstall ovirt-vmconsole-host-1.0.9-3.el9.noarch and ovirt- vmconsole-1.0.9-3.el9.noarch , but with no success, the same situation with reinstall virt-viewer 11.0 locally ( ubuntu 23.04 last updates ). This stopped my VMs deploying :( . Exists any fix or workaround about this issue ?? thx. a lot regs. Pavel

1 0

Re: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Strahil Nikolov 18 Jan '24

18 Jan '24

Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you need to disable the fencing or prevent it somehow - like changing the ipmi user's password).The engine itself uses ssh to distribute the new certificate to the host. Best Regards,Strahil Nikolov On Wed, Jan 17, 2024 at 18:07, Michael Douglass<mikedoug(a)certida.com> wrote: Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users. From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Sent: Saturday, January 13, 2024 6:57 PM To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users <users(a)ovirt.org> Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida.Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment.The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards,Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users<users(a)ovirt.org> wrote:This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEAND…

2 1

gluster on iSCSI devices in ovirt environment
by paf1＠email.cz 18 Jan '24

18 Jan '24

hello dears, can anybody explain me HOWTO realize 2 nodes + aribiter gluster from two (three) locations on block iSCSI devices ? Something like this: gluster volume create TEST replica 3 arbiter 1 <location-one-host1 - iSCSI target > <location-two-host2 - iSCSI target> < location-three-host3 - /dev/sda5 e.g. > - ALL applied on multinode ovirt cluster thx a lot for any help regs. Pa.

3 4

NVIDIA vGPU driver for Ovirt 4.5.4
by michael.a.silveira3.ctr＠us.navy.mil 18 Jan '24

18 Jan '24

Hello, Does anyone know which, if any, NVIDIA GRID driver supports Ovirt 4.5.4 on Ovirt-node (kernel 4.18.0-408.el8.x86_64)? I've recently upgraded to Ovirt 4.5 and can't find a NVIDIA GRID driver that will connect to my Tesla v100 on the new kernel. nvidia-smi returns the following no matter what driver I install: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

5 4

Cannot restart ovirt after massive failure.
by Gilboa Davara 18 Jan '24

18 Jan '24

Hello all, During the night, one of my (smaller) setups, a single node self hosted engine (localhost NFS) crashed due to what-looks-like a massive disk failure (Software RAID6, with 10 drives + spare). After a reboot, I let the RAID resync with a fresh drive) and went on to start oVirt. However, no such luck. Two issues: 1. ovirt-ha-broker fails due to broken hosted engine state (log attached). 2. ovirt-ha-agent fails due to network test (tcp) even though both remote-host and DNS servers are active. (log attached). Two questions: 1. Can I somehow force the agent to disable the network liveliness test? 2. Can I somehow force the broker to rebuild / fix the hosted engine state? - Gilboa

4 17

Cannot open Vm Console
by Vittorio 18 Jan '24

18 Jan '24

I got few Vm on my ovirte nodes, but i cant acess to console. When i downloaded it, the error is the following : "Failed to complete haandshake Errore in the pull function."

2 1