- Users - oVirt List Archives

Re: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Strahil Nikolov 18 Jan '24

18 Jan '24

Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you need to disable the fencing or prevent it somehow - like changing the ipmi user's password).The engine itself uses ssh to distribute the new certificate to the host. Best Regards,Strahil Nikolov On Wed, Jan 17, 2024 at 18:07, Michael Douglass<mikedoug(a)certida.com> wrote: Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users. From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Sent: Saturday, January 13, 2024 6:57 PM To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users <users(a)ovirt.org> Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida.Hi, Recently I found another way to renew the certificates on a one-node self-hosted environment.The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again. Best Regards,Strahil Nikolov On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users<users(a)ovirt.org> wrote:This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon! The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future. If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work. Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :) ########### On the ENGINE HOST ########### ## Check the CA Cert on the ENGINE openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates ## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script: ## NOTE: these are steps from another post I didn't need to do them -- could be dangerous... ``` set -x ## Make the script echo everything out, so if it fails you know where set -e ## Make the script STOP on any error set my_date="$(date +"%Y%m%d%H%M%S")" # Backup the existing CA files /bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date /bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date} /bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date} # Sign the key openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256 openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full # Put the files into place /bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem /bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der ``` Now you need to copy the new CA file over to the host: Source: ENGINE `/etc/pki/ovirt-engine/ca.pem` Dest: HOST `/tmp/new-ca.pem` ########### On the oVirt Host ########### # Create a CSR using the information from the existing certificate and the existing key: openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem Now you need to copy the new CA file over to the host: Source: HOST `/tmp/HOST.csr` Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req` ########### On the ENGINE HOST ########### # Now sign it: /usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com # NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names) Now you need to copy the new Certificate file over to the host: Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer Dest: HOST /tmp/new-cert.pem ########### On the oVirt Host ########### Run this script to put the cert and CA in place. Note if you don't put a ca into /tmp/new-ca.pem it skips that step. ``` set -x set -e set my_date="$(date +"%Y%m%d%H%M%S")" for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-cert.pem ${x} chmod 644 ${x} chown root:kvm ${x} done if -f /tmp/new-ca.pem; then for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do /bin/mv -n $x ${x}.${mydate} /bin/cp /tmp/new-ca.pem ${x} chmod 644 ${x} chown root:kvm ${x} done fi ``` Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again! # Restart the two services affected by the key/cert changes systemctl restart vdsmd libvirtd _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEAND…

2 1

gluster on iSCSI devices in ovirt environment
by paf1＠email.cz 18 Jan '24

18 Jan '24

hello dears, can anybody explain me HOWTO realize 2 nodes + aribiter gluster from two (three) locations on block iSCSI devices ? Something like this: gluster volume create TEST replica 3 arbiter 1 <location-one-host1 - iSCSI target > <location-two-host2 - iSCSI target> < location-three-host3 - /dev/sda5 e.g. > - ALL applied on multinode ovirt cluster thx a lot for any help regs. Pa.

3 4

NVIDIA vGPU driver for Ovirt 4.5.4
by michael.a.silveira3.ctr＠us.navy.mil 18 Jan '24

18 Jan '24

Hello, Does anyone know which, if any, NVIDIA GRID driver supports Ovirt 4.5.4 on Ovirt-node (kernel 4.18.0-408.el8.x86_64)? I've recently upgraded to Ovirt 4.5 and can't find a NVIDIA GRID driver that will connect to my Tesla v100 on the new kernel. nvidia-smi returns the following no matter what driver I install: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

5 4

Cannot restart ovirt after massive failure.
by Gilboa Davara 18 Jan '24

18 Jan '24

Hello all, During the night, one of my (smaller) setups, a single node self hosted engine (localhost NFS) crashed due to what-looks-like a massive disk failure (Software RAID6, with 10 drives + spare). After a reboot, I let the RAID resync with a fresh drive) and went on to start oVirt. However, no such luck. Two issues: 1. ovirt-ha-broker fails due to broken hosted engine state (log attached). 2. ovirt-ha-agent fails due to network test (tcp) even though both remote-host and DNS servers are active. (log attached). Two questions: 1. Can I somehow force the agent to disable the network liveliness test? 2. Can I somehow force the broker to rebuild / fix the hosted engine state? - Gilboa

4 17

Cannot open Vm Console
by Vittorio 18 Jan '24

18 Jan '24

I got few Vm on my ovirte nodes, but i cant acess to console. When i downloaded it, the error is the following : "Failed to complete haandshake Errore in the pull function."

2 1

VDSM Command DetachStorageDomainVDS Failed
by Matthew J Black 18 Jan '24

18 Jan '24

Hi All, So after having an oVirt cluster crash (no data loss (at this stage), thankfully) and rebuilding from scratch, I'm trying to import the old Storage Domains. I've been successful with three (automatically "detaching" from the old, now non-existent hosts) but one is giving me the following error from the hosted-engine GUI: ~~~ VDSM command DetachStorageDomainVDS failed: Cannot acquire host id: ('e311ddf1-7f2c-49ef-a618-050d9a2b947f', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) ~~~ I am going to assume ("make an ass out for you and me") that I can run the required command from the cli with a "-force" flag (I hope), so my Q is: What is the command, because I can't seem to find it in any doco (most probably because I'm old and my eyes are feeble :-) ). Any help gratefully appreciated - FTR: that particular Storage Domain have a bunch of VM Images on it which I'd rather import/recover than have to create from scratch. Cheers Dulux-Oz

1 0

Deployment Error: Host is not up - Looking For Some Advice/Pointers
by Matthew J Black 18 Jan '24

18 Jan '24

Hi All, So, on a fresh install on RL v9.3, we're getting a `Host is not up, please check logs, perhaps also on the engine machine` error. This comes up right after the 20min timeout (the 120sec * 10 one). No, the hosted-engine is not deployed (ie hosted-engine --check-deployed). Obviously I need to check the logs, but which ones in particular (the hosted-engine-setup logs, obviously, but which other ones), where are they if the hosted-engine is not running, and, in an effort to narrow down the volume of info to a more manageable level, what should I be looking for? Thanks in advance

1 1

Please, Please Help - New oVirt Install/Deployment Failing - "Host is not up..."
by Matthew J Black 18 Jan '24

18 Jan '24

Hi Everyone, Could someone please help me - I've been trying to do an install of oVirt for *weeks* (including false starts and self-inflicted wounds/errors) and it is still not working. My setup: - oVirt v4.5.3 - A brand new fresh vanilla install of RockyLinux 8.6 - all working AOK - 2*NICs in a bond (802.3ad) with a couple of sub-Interfaces/VLANs - all working AOK - All relevant IPv4 Address in DNS with Reverse Lookups - all working AOK - All relevant IPv4 Address in "/etc/hosts" file - all working AOK - IPv6 (using "method=auto" in the interface config file) enabled on the relevant sub-Interface/VLAN - I'm not using IPv6 on the network, only IPv4, but I'm trying to cover all the bases. - All relevant Ports (as per the oVirt documentation) set up on the firewall - ie firewall-cmd --add-service={{ libvirt-tls | ovirt-imageio | ovirt-vmconsole | vdsm }} - All the relevant Repositories installed (ie RockyLinux BaseOS, AppStream, & PowerTools, and the EPEL, plus the ones from the oVirt documentation) I have followed the oVirt documentation (including the special RHEL-instructions and RockyLinux-instructions) to the letter - no deviations, no special settings, exactly as they are written. All the dnf installs, etc, went off without a hitch, including the "dnf install centos-release-ovirt45", "dnf install ovirt-engine-appliance", and "dnf install ovirt-hosted-engine-setup" - no errors anywhere. Here is the results of a "dnf repolist": - appstream Rocky Linux 8 - AppStream - baseos Rocky Linux 8 - BaseOS - centos-ceph-pacific CentOS-8-stream - Ceph Pacific - centos-gluster10 CentOS-8-stream - Gluster 10 - centos-nfv-openvswitch CentOS-8 - NFV OpenvSwitch - centos-opstools CentOS-OpsTools - collectd - centos-ovirt45 CentOS Stream 8 - oVirt 4.5 - cs8-extras CentOS Stream 8 - Extras - cs8-extras-common CentOS Stream 8 - Extras common packages - epel Extra Packages for Enterprise Linux 8 - x86_64 - epel-modular Extra Packages for Enterprise Linux Modular 8 - x86_64 - ovirt-45-centos-stream-openstack-yoga CentOS Stream 8 - oVirt 4.5 - OpenStack Yoga Repository - ovirt-45-upstream oVirt upstream for CentOS Stream 8 - oVirt 4.5 - powertools Rocky Linux 8 - PowerTools So I kicked-off the oVirt deployment with: "hosted-engine --deploy --4 --ansible-extra-vars=he_offline_deployment=true". I used "--ansible-extra-vars=he_offline_deployment=true" because without that flag I was getting "DNF timout" issues (see my previous post `Local (Deployment) VM Can't Reach "centos-ceph-pacific" Repo`). I answer the defaults to all of questions the script asked, or entered the deployment-relevant answers where appropriate. In doing this I double-checked every answer before hitting <Enter>. Everything progressed smoothly until the deployment reached the "Wait for the host to be up" task... which then hung for more than 30 minutes before failing. From the ovirt-hosted-engine-setup... log file: - 2022-10-20 17:54:26,285+1100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} I checked the following log files and found all of the relevant ERROR lines, then checked several 10s of proceeding and succeeding lines trying to determine what was going wrong, but I could not determine anything. - ovirt-hosted-engine-setup... - ovirt-hosted-engine-setup-ansible-bootstrap_local_vm... - ovirt-hosted-engine-setup-ansible-final_clean... - not really relevant, I believe I can include the log files (or the relevant parts of the log files) if people want - but that are very large: several 100 kilobytes each. I also googled "oVirt Host is not up" and found several entries, but after reading them all the most relevant seems to be a thread from these mailing list: `Install of RHV 4.4 failing - "Host is not up, please check logs, perhaps also on the engine machine"` - but this seems to be talking about an upgrade and I didn't gleam anything useful from it - I could, of course, be wrong about that. So my questions are: - Where else should I be looking (ie other log files, etc, and possible where to find them)? - Does anyone have any idea why this isn't working? - Does anyone have a work-around (including a completely manual process to get things working - I don't mind working in the CLI with virsh, etc)? - What am I doing wrong? Please, I'm really stumped with this, and I really do need help. Cheers Dulux-Oz

8 22

ovirt node ng 4.5.5 fresh install fails
by Levi Wilbert 17 Jan '24

17 Jan '24

I'm attempting to update our oVirt cluster to 4.5.5 from 4.5.4, running oVirt Node NG on the hosts. When I tried updating a host through the oVirt Manager GUI, after the host reboots, it fails to start up and goes into emergency recovery mode: [ 4.534872] localhost systemd[1]: Reached target Local File Systems. [ 4.535119] localhost systemd[1]: Reached target System Initialization. [ 4.535343] localhost systemd[1]: Reached target Basic System. [ 4.536759] localhost systemd[1]: Started Hardware RNG Entropy Gatherer Daemon. [ 4.541801] localhost rngd[1512]: Disabling 7: PKCS11 Entropy generator (pkcs11) [ 4.541801] localhost rngd[1512]: Disabling 5: NIST Network Entropy Beacon (nist) [ 4.541801] localhost rngd[1512]: Disabling 9: Qrypt quantum entropy beacon (qrypt) [ 4.541801] localhost rngd[1512]: Initializing available sources [ 4.542073] localhost rngd[1512]: [hwrng ]: Initialization Failed [ 4.542073] localhost rngd[1512]: [rdrand]: Enabling RDSEED rng support [ 4.542073] localhost rngd[1512]: [rdrand]: Initialized [ 4.542073] localhost rngd[1512]: [jitter]: JITTER timeout set to 5 sec [ 4.582381] localhost rngd[1512]: [jitter]: Initializing AES buffer [ 8.309063] localhost rngd[1512]: [jitter]: Enabling JITTER rng support [ 8.309063] localhost rngd[1512]: [jitter]: Initialized [ 133.884355] localhost dracut-initqueue[1095]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks: [ 133.885349] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-3f47cad8:fecb96ea:0ea37615:4e5dec4e.sh: "[ -e "/dev/disk/by-id/md-uuid-3f47cad8:fecb96ea:0ea37615:4e5dec4e" ]" [ 133.886485] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-d446b801:d515c112:116ff07f:9ae52466.sh: "[ -e "/dev/disk/by-id/md-uuid-d446b801:d515c112:116ff07f:9ae52466" ]" [ 133.887619] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fonn\x2fovirt-node-ng-4.5.5-0.20231130.0+1.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup(a)*.service 2>/dev/null; then [ 133.887619] localhost dracut-initqueue[1095]: [ -e "/dev/onn/ovirt-node-ng-4.5.5-0.20231130.0+1" ] [ 133.887619] localhost dracut-initqueue[1095]: fi" [ 133.888667] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fonn\x2fswap.sh: "[ -e "/dev/onn/swap" ]" [ 133.890050] localhost dracut-initqueue[1095]: Warning: dracut-initqueue: starting timeout scripts [ 133.969228] localhost dracut-initqueue[7366]: Scanning devices md126p2 for LVM logical volumes onn/ovirt-node-ng-4.5.5-0.20231130.0+1 [ 133.969228] localhost dracut-initqueue[7366]: onn/swap [ 134.001560] localhost dracut-initqueue[7366]: onn/ovirt-node-ng-4.5.5-0.20231130.0+1 thin [ 134.001560] localhost dracut-initqueue[7366]: onn/swap linear [ 134.014259] localhost dracut-initqueue[7381]: /etc/lvm/profile/imgbased-pool.profile: stat failed: No such file or directory [ 134.532608] localhost dracut-initqueue[7381]: Check of pool onn/pool00 failed (status:64). Manual repair required! I then attempted installing the oVirt Node NG 4.5.5 iso to a USB stick and tried installing that way, however, after going through the GUI and setting up storage, network, hostname, etc, the install fails shortly after clicking "Begin". 22:11:32,671 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:blivet:executing action: [468] destroy device lvmthinlv onn-var_log_audit (id 216) 22:11:32,672 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMLogicalVolumeDevice.destroy: onn-var_log_audit ; status: False ; 22:11:32,673 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMLogicalVolumeDevice.teardown: onn-var_log_audit ; status: False ; controllable: False ; 22:11:32,674 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMVolumeGroupDevice.setup_parents: name: onn ; orig: True ; 22:11:32,674 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: PartitionDevice.setup: Volume0_0p2 ; orig: True ; status: True ; controllable: True ; 22:11:32,675 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMPhysicalVolume.setup: device: /dev/md/Volume0_0p2 ; type: lvmpv ; status: False ; 22:11:32,676 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMLogicalVolumeDevice._destroy: onn-var_log_audit ; status: False ; 22:11:32,676 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running [97] lvm lvremove --yes onn/var_log_audit --config= log {level=7 file=/tmp/lvm.log syslog=0} --devices=/dev/md/Volume0_0p2 ... 22:11:33,104 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ] 22:11:33,105 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ] 22:11:33,105 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ] 22:11:33,106 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ] 22:11:33,106 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ] 22:11:33,107 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ] 22:11:33,309 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stdout[97]: 22:11:33,310 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stderr[97]: /etc/lvm/profile/imgbased-pool.profile: stat failed: No such file or directory 22:11:33,310 WARNING org.fedoraproject.Anaconda.Modules.Storage: Check of pool onn/pool00 failed (status:64). Manual repair required! I'm wondering if it has to do with installing oVirt node on a RAID mirror?

1 0

Migrating from a self hosted engine to standalone
by redhat＠intheoutback.com 17 Jan '24

17 Jan '24

Hi, I am in the process of moving our oVirt environment from a self hosted engine to a standalone engine on it's own HW. I have Googled and found a procedure for standalone > self hosted, but not the other way around. My current situation is that I have 5 locations running oVirt 4.3 with 3 to 4 hypervisors and iSCSI storage backend, with a self hosted engine. All these locations are operational and short a downtime is acceptable if a must, losing a VM is NOT good. I also have 1 oVirt environment 4.3 with 2 hypervisors and iSCSI backend that is my QA/Test. All my networks are on internal networks with no outside world connections. Most importantly, we are also looking at upgrading from 4.3 to 4.4 I have not found any straightforward way to migrate from self hosted to standalone. My current plan is to do the following. 1) Create a new 4.4 standalone engine 2) Remove one hypervisor from the 4.3 cluster 3) Kickstart the hypervisor to RHEL 8.8 and configure ready for oVirt 4.4 4) Add the new host to the standalone engine. 5) Shutdown and export a number of VMs in the oVirt 4.3 and import them in to the new oVirt 4.4. 6) Repeat steps 2 > 5 until the everything is moved over. Just wanting to get your expert opinions on this method or is there a much quicker easier method that will not risk the chances of losing the cluster/VMs or an extended outage. Since we need to upgrade anyway from 4.3 to 4.4 I thought this the better method that upgrading the operational clusters. Thanks

4 3