Re: [EXT]Re: Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Strahil Nikolov
Actually , the vdsmd trick is needed only to fool the engine that the host has failed (you need to disable the fencing or prevent it somehow - like changing the ipmi user's password).The engine itself uses ssh to distribute the new certificate to the host.
Best Regards,Strahil Nikolov
On Wed, Jan 17, 2024 at 18:07, Michael Douglass<mikedoug(a)certida.com> wrote: Ah, the steps I list here work for non-single node environments as well. The problem with vdsm-tool doing the certificate renewal in that setup is that the certificate needs to be signed by the engine host and not the local host. Apologies if this causes any confusion for single-node users.
From: Strahil Nikolov <hunter86_bg(a)yahoo.com>
Sent: Saturday, January 13, 2024 6:57 PM
To: Michael Douglass <mikedoug(a)certida.com>; mikedoug--- via Users <users(a)ovirt.org>
Subject: [EXT]Re: [ovirt-users] Re: [SOLVED] Re: Re: How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment? CAUTION: Be careful of links - This email originated from outside of Certida.Hi,
Recently I found another way to renew the certificates on a one-node self-hosted environment.The trick is to stop vdsmd and wait till the engine shows the system unresponsive. Then you can trigger a certificate renewal and just power on vdsmd again.
Best Regards,Strahil Nikolov
On Sun, Jan 14, 2024 at 2:54, mikedoug--- via Users<users(a)ovirt.org> wrote:This is the only place we found the answer we needed concerning how to sign a host key from the engine! Saved our bacon!
The only tweak I would make is that there were a few more destinations at the end to copy the files into, and instead of rebooting the host we found that just restarting vdsmd and libvirtd got everything working without any existing VMs having to stop. Here's a complete update of what you have above intermingling some extra bits for anyone who has to do this in the future.
If the certs on your oVirt host expire, it can be a PITA to figure out how to fix it. It's actually simple, but takes a LOT of manual work.
Make sure every part of every script makes sense as I made some modifications as I documented it! I do not warrant that I didn't make a mistake somewhere. :)
########### On the ENGINE HOST ###########
## Check the CA Cert on the ENGINE
openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates
## If it is expired, then on the ENGINE, rebuild the CA Cert with this in a script:
## NOTE: these are steps from another post I didn't need to do them -- could be dangerous...
```
set -x ## Make the script echo everything out, so if it fails you know where
set -e ## Make the script STOP on any error
set my_date="$(date +"%Y%m%d%H%M%S")"
# Backup the existing CA files
/bin/cp -p /etc/pki/ovirt-engine/private/ca.pem /etc/pki/ovirt-engine/private/ca.pem.$my_date
/bin/cp -p /etc/pki/ovirt-engine/ca.pem{,.$my_date}
/bin/mv /etc/pki/ovirt-engine/certs/ca.der{,.$my_date}
# Sign the key
openssl x509 -signkey /etc/pki/ovirt-engine/private/ca.pem -in /etc/pki/ovirt-engine/ca.pem -out /etc/pki/ovirt-engine/ca.pem.new -days 3650 -sha256
openssl x509 -in /etc/pki/ovirt-engine/ca.pem.new -text > /etc/pki/ovirt-engine/ca.pem.new.full
# Put the files into place
/bin/mv -f /etc/pki/ovirt-engine/ca.pem.new.full /etc/pki/ovirt-engine/ca.pem
/bin/cp -p /etc/pki/ovirt-engine/ca.pem.new /etc/pki/ovirt-engine/certs/ca.der
```
Now you need to copy the new CA file over to the host:
Source: ENGINE `/etc/pki/ovirt-engine/ca.pem`
Dest: HOST `/tmp/new-ca.pem`
########### On the oVirt Host ###########
# Create a CSR using the information from the existing certificate and the existing key:
openssl x509 -x509toreq -in /etc/pki/libvirt/clientcert.pem -out /tmp/HOST.csr -signkey /etc/pki/libvirt/private/clientkey.pem
Now you need to copy the new CA file over to the host:
Source: HOST `/tmp/HOST.csr`
Dest: ENGINE `/etc/pki/ovirt-engine/requests/full.hostname.com.req`
########### On the ENGINE HOST ###########
# Now sign it:
/usr/share/ovirt-engine/bin/pki-enroll-request.sh --name=full.hostname.com
# NB -- adding --san results in an error: --san=host.na.me (So no Subject Alternate Names)
Now you need to copy the new Certificate file over to the host:
Source: ENGINE /etc/pki/ovirt-engine/certs/full.hostname.com.cer
Dest: HOST /tmp/new-cert.pem
########### On the oVirt Host ###########
Run this script to put the cert and CA in place. Note if you don't put a ca into
/tmp/new-ca.pem it skips that step.
```
set -x
set -e
set my_date="$(date +"%Y%m%d%H%M%S")"
for x in /etc/pki/libvirt/clientcert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem; do
/bin/mv -n $x ${x}.${mydate}
/bin/cp /tmp/new-cert.pem ${x}
chmod 644 ${x}
chown root:kvm ${x}
done
if -f /tmp/new-ca.pem; then
for x in /etc/pki/vdsm/libvirt-migrate/ca-cert.pem /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem /etc/pki/CA/cacert.pem; do
/bin/mv -n $x ${x}.${mydate}
/bin/cp /tmp/new-ca.pem ${x}
chmod 644 ${x}
chown root:kvm ${x}
done
fi
```
Now you're ready to restart two vital services -- some people say "reboot the host" -- but we found that unecessary. Running this restart was safe for us and didn't cause any running VMs to crash or reboot -- they kept going without issue. Once we did this, waited a few minutes, the host came back up on the engine and everything was happy. Specifically a VM we were unable to get running was a click away from full functionality again!
# Restart the two services affected by the key/cert changes
systemctl restart vdsmd libvirtd
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2EBSBWXFCEA...
3 months, 3 weeks
gluster on iSCSI devices in ovirt environment
by paf1＠email.cz
hello dears,
can anybody explain me HOWTO realize 2 nodes + aribiter gluster from
two (three) locations on block iSCSI devices ?
Something like this:
gluster volume create TEST replica 3 arbiter 1 <location-one-host1 -
iSCSI target > <location-two-host2 - iSCSI target> <
location-three-host3 - /dev/sda5 e.g. > - ALL applied on multinode
ovirt cluster
thx a lot for any help
regs.
Pa.
3 months, 3 weeks
NVIDIA vGPU driver for Ovirt 4.5.4
by michael.a.silveira3.ctr＠us.navy.mil
Hello,
Does anyone know which, if any, NVIDIA GRID driver supports Ovirt 4.5.4 on Ovirt-node (kernel 4.18.0-408.el8.x86_64)? I've recently upgraded to Ovirt 4.5 and can't find a NVIDIA GRID driver that will connect to my Tesla v100 on the new kernel. nvidia-smi returns the following no matter what driver I install:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
3 months, 3 weeks
Cannot restart ovirt after massive failure.
by Gilboa Davara
Hello all,
During the night, one of my (smaller) setups, a single node self hosted
engine (localhost NFS) crashed due to what-looks-like a massive disk
failure (Software RAID6, with 10 drives + spare).
After a reboot, I let the RAID resync with a fresh drive) and went on to
start oVirt.
However, no such luck.
Two issues:
1. ovirt-ha-broker fails due to broken hosted engine state (log attached).
2. ovirt-ha-agent fails due to network test (tcp) even though both
remote-host and DNS servers are active. (log attached).
Two questions:
1. Can I somehow force the agent to disable the network liveliness test?
2. Can I somehow force the broker to rebuild / fix the hosted engine state?
- Gilboa
3 months, 3 weeks
Cannot open Vm Console
by Vittorio
I got few Vm on my ovirte nodes, but i cant acess to console.
When i downloaded it, the error is the following :
"Failed to complete haandshake Errore in the pull function."
3 months, 3 weeks
VDSM Command DetachStorageDomainVDS Failed
by Matthew J Black
Hi All,
So after having an oVirt cluster crash (no data loss (at this stage), thankfully) and rebuilding from scratch, I'm trying to import the old Storage Domains. I've been successful with three (automatically "detaching" from the old, now non-existent hosts) but one is giving me the following error from the hosted-engine GUI:
~~~
VDSM command DetachStorageDomainVDS failed: Cannot acquire host id: ('e311ddf1-7f2c-49ef-a618-050d9a2b947f', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))
~~~
I am going to assume ("make an ass out for you and me") that I can run the required command from the cli with a "-force" flag (I hope), so my Q is: What is the command, because I can't seem to find it in any doco (most probably because I'm old and my eyes are feeble :-) ).
Any help gratefully appreciated - FTR: that particular Storage Domain have a bunch of VM Images on it which I'd rather import/recover than have to create from scratch.
Cheers
Dulux-Oz
3 months, 3 weeks
Deployment Error: Host is not up - Looking For Some Advice/Pointers
by Matthew J Black
Hi All,
So, on a fresh install on RL v9.3, we're getting a `Host is not up, please check logs, perhaps also on the engine machine` error.
This comes up right after the 20min timeout (the 120sec * 10 one).
No, the hosted-engine is not deployed (ie hosted-engine --check-deployed).
Obviously I need to check the logs, but which ones in particular (the hosted-engine-setup logs, obviously, but which other ones), where are they if the hosted-engine is not running, and, in an effort to narrow down the volume of info to a more manageable level, what should I be looking for?
Thanks in advance
3 months, 3 weeks
Please, Please Help - New oVirt Install/Deployment Failing - "Host is not up..."
by Matthew J Black
Hi Everyone,
Could someone please help me - I've been trying to do an install of oVirt for *weeks* (including false starts and self-inflicted wounds/errors) and it is still not working.
My setup:
- oVirt v4.5.3
- A brand new fresh vanilla install of RockyLinux 8.6 - all working AOK
- 2*NICs in a bond (802.3ad) with a couple of sub-Interfaces/VLANs - all working AOK
- All relevant IPv4 Address in DNS with Reverse Lookups - all working AOK
- All relevant IPv4 Address in "/etc/hosts" file - all working AOK
- IPv6 (using "method=auto" in the interface config file) enabled on the relevant sub-Interface/VLAN - I'm not using IPv6 on the network, only IPv4, but I'm trying to cover all the bases.
- All relevant Ports (as per the oVirt documentation) set up on the firewall
- ie firewall-cmd --add-service={{ libvirt-tls | ovirt-imageio | ovirt-vmconsole | vdsm }}
- All the relevant Repositories installed (ie RockyLinux BaseOS, AppStream, & PowerTools, and the EPEL, plus the ones from the oVirt documentation)
I have followed the oVirt documentation (including the special RHEL-instructions and RockyLinux-instructions) to the letter - no deviations, no special settings, exactly as they are written.
All the dnf installs, etc, went off without a hitch, including the "dnf install centos-release-ovirt45", "dnf install ovirt-engine-appliance", and "dnf install ovirt-hosted-engine-setup" - no errors anywhere.
Here is the results of a "dnf repolist":
- appstream Rocky Linux 8 - AppStream
- baseos Rocky Linux 8 - BaseOS
- centos-ceph-pacific CentOS-8-stream - Ceph Pacific
- centos-gluster10 CentOS-8-stream - Gluster 10
- centos-nfv-openvswitch CentOS-8 - NFV OpenvSwitch
- centos-opstools CentOS-OpsTools - collectd
- centos-ovirt45 CentOS Stream 8 - oVirt 4.5
- cs8-extras CentOS Stream 8 - Extras
- cs8-extras-common CentOS Stream 8 - Extras common packages
- epel Extra Packages for Enterprise Linux 8 - x86_64
- epel-modular Extra Packages for Enterprise Linux Modular 8 - x86_64
- ovirt-45-centos-stream-openstack-yoga CentOS Stream 8 - oVirt 4.5 - OpenStack Yoga Repository
- ovirt-45-upstream oVirt upstream for CentOS Stream 8 - oVirt 4.5
- powertools Rocky Linux 8 - PowerTools
So I kicked-off the oVirt deployment with: "hosted-engine --deploy --4 --ansible-extra-vars=he_offline_deployment=true".
I used "--ansible-extra-vars=he_offline_deployment=true" because without that flag I was getting "DNF timout" issues (see my previous post `Local (Deployment) VM Can't Reach "centos-ceph-pacific" Repo`).
I answer the defaults to all of questions the script asked, or entered the deployment-relevant answers where appropriate. In doing this I double-checked every answer before hitting <Enter>. Everything progressed smoothly until the deployment reached the "Wait for the host to be up" task... which then hung for more than 30 minutes before failing.
From the ovirt-hosted-engine-setup... log file:
- 2022-10-20 17:54:26,285+1100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"}
I checked the following log files and found all of the relevant ERROR lines, then checked several 10s of proceeding and succeeding lines trying to determine what was going wrong, but I could not determine anything.
- ovirt-hosted-engine-setup...
- ovirt-hosted-engine-setup-ansible-bootstrap_local_vm...
- ovirt-hosted-engine-setup-ansible-final_clean... - not really relevant, I believe
I can include the log files (or the relevant parts of the log files) if people want - but that are very large: several 100 kilobytes each.
I also googled "oVirt Host is not up" and found several entries, but after reading them all the most relevant seems to be a thread from these mailing list: `Install of RHV 4.4 failing - "Host is not up, please check logs, perhaps also on the engine machine"` - but this seems to be talking about an upgrade and I didn't gleam anything useful from it - I could, of course, be wrong about that.
So my questions are:
- Where else should I be looking (ie other log files, etc, and possible where to find them)?
- Does anyone have any idea why this isn't working?
- Does anyone have a work-around (including a completely manual process to get things working - I don't mind working in the CLI with virsh, etc)?
- What am I doing wrong?
Please, I'm really stumped with this, and I really do need help.
Cheers
Dulux-Oz
3 months, 3 weeks
ovirt node ng 4.5.5 fresh install fails
by Levi Wilbert
I'm attempting to update our oVirt cluster to 4.5.5 from 4.5.4, running oVirt Node NG on the hosts.
When I tried updating a host through the oVirt Manager GUI, after the host reboots, it fails to start up and goes into emergency recovery mode:
[ 4.534872] localhost systemd[1]: Reached target Local File Systems.
[ 4.535119] localhost systemd[1]: Reached target System Initialization.
[ 4.535343] localhost systemd[1]: Reached target Basic System.
[ 4.536759] localhost systemd[1]: Started Hardware RNG Entropy Gatherer Daemon.
[ 4.541801] localhost rngd[1512]: Disabling 7: PKCS11 Entropy generator (pkcs11)
[ 4.541801] localhost rngd[1512]: Disabling 5: NIST Network Entropy Beacon (nist)
[ 4.541801] localhost rngd[1512]: Disabling 9: Qrypt quantum entropy beacon (qrypt)
[ 4.541801] localhost rngd[1512]: Initializing available sources
[ 4.542073] localhost rngd[1512]: [hwrng ]: Initialization Failed
[ 4.542073] localhost rngd[1512]: [rdrand]: Enabling RDSEED rng support
[ 4.542073] localhost rngd[1512]: [rdrand]: Initialized
[ 4.542073] localhost rngd[1512]: [jitter]: JITTER timeout set to 5 sec
[ 4.582381] localhost rngd[1512]: [jitter]: Initializing AES buffer
[ 8.309063] localhost rngd[1512]: [jitter]: Enabling JITTER rng support
[ 8.309063] localhost rngd[1512]: [jitter]: Initialized
[ 133.884355] localhost dracut-initqueue[1095]: Warning: dracut-initqueue: timeout, still waiting for following initqueue hooks:
[ 133.885349] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-3f47cad8:fecb96ea:0ea37615:4e5dec4e.sh: "[ -e "/dev/disk/by-id/md-uuid-3f47cad8:fecb96ea:0ea37615:4e5dec4e" ]"
[ 133.886485] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fdisk\x2fby-id\x2fmd-uuid-d446b801:d515c112:116ff07f:9ae52466.sh: "[ -e "/dev/disk/by-id/md-uuid-d446b801:d515c112:116ff07f:9ae52466" ]"
[ 133.887619] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fonn\x2fovirt-node-ng-4.5.5-0.20231130.0+1.sh: "if ! grep -q After=remote-fs-pre.target /run/systemd/generator/systemd-cryptsetup(a)*.service 2>/dev/null; then
[ 133.887619] localhost dracut-initqueue[1095]: [ -e "/dev/onn/ovirt-node-ng-4.5.5-0.20231130.0+1" ]
[ 133.887619] localhost dracut-initqueue[1095]: fi"
[ 133.888667] localhost dracut-initqueue[1095]: Warning: /lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2fonn\x2fswap.sh: "[ -e "/dev/onn/swap" ]"
[ 133.890050] localhost dracut-initqueue[1095]: Warning: dracut-initqueue: starting timeout scripts
[ 133.969228] localhost dracut-initqueue[7366]: Scanning devices md126p2 for LVM logical volumes onn/ovirt-node-ng-4.5.5-0.20231130.0+1
[ 133.969228] localhost dracut-initqueue[7366]: onn/swap
[ 134.001560] localhost dracut-initqueue[7366]: onn/ovirt-node-ng-4.5.5-0.20231130.0+1 thin
[ 134.001560] localhost dracut-initqueue[7366]: onn/swap linear
[ 134.014259] localhost dracut-initqueue[7381]: /etc/lvm/profile/imgbased-pool.profile: stat failed: No such file or directory
[ 134.532608] localhost dracut-initqueue[7381]: Check of pool onn/pool00 failed (status:64). Manual repair required!
I then attempted installing the oVirt Node NG 4.5.5 iso to a USB stick and tried installing that way, however, after going through the GUI and setting up storage, network, hostname, etc, the install fails shortly after clicking "Begin".
22:11:32,671 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:blivet:executing action: [468] destroy device lvmthinlv onn-var_log_audit (id 216)
22:11:32,672 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMLogicalVolumeDevice.destroy: onn-var_log_audit ; status: False ;
22:11:32,673 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMLogicalVolumeDevice.teardown: onn-var_log_audit ; status: False ; controllable: False ;
22:11:32,674 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMVolumeGroupDevice.setup_parents: name: onn ; orig: True ;
22:11:32,674 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: PartitionDevice.setup: Volume0_0p2 ; orig: True ; status: True ; controllable: True ;
22:11:32,675 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMPhysicalVolume.setup: device: /dev/md/Volume0_0p2 ; type: lvmpv ; status: False ;
22:11:32,676 WARNING org.fedoraproject.Anaconda.Modules.Storage:DEBUG:blivet: LVMLogicalVolumeDevice._destroy: onn-var_log_audit ; status: False ;
22:11:32,676 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running [97] lvm lvremove --yes onn/var_log_audit --config= log {level=7 file=/tmp/lvm.log syslog=0} --devices=/dev/md/Volume0_0p2 ...
22:11:33,104 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ]
22:11:33,105 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ]
22:11:33,105 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ]
22:11:33,106 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ]
22:11:33,106 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ]
22:11:33,107 ERR rsyslogd:imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-3.el9 try https://www.rsyslog.com/e/2433 ]
22:11:33,309 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stdout[97]:
22:11:33,310 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stderr[97]: /etc/lvm/profile/imgbased-pool.profile: stat failed: No such file or directory
22:11:33,310 WARNING org.fedoraproject.Anaconda.Modules.Storage: Check of pool onn/pool00 failed (status:64). Manual repair required!
I'm wondering if it has to do with installing oVirt node on a RAID mirror?
3 months, 3 weeks
How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Derek Atkins
Hi,
I've got a single-host hosted-engine deployment that I originally
installed with 4.0 and have upgraded over the years to 4.3.10. I and some
of my users have upgraded remote-viewer and now I get an error when I try
to view the console of my VMs:
(remote-viewer:8252): Spice-WARNING **: 11:30:41.806:
../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error
in server certificate verification: CA signature digest algorithm too weak
(num=68:depth0:/O=<My Org Name>/CN=<Host's Name>)
I am 99.99% sure this is because the old certs use SHA1.
I reran engine-setup on the engine and it asked me if I wanted to renew
the PKI, and I answered yes. This replaced many[1] of the certificates in
/etc/pki/ovirt-engine/certs on the engine, but it did not update the
Host's certificate.
All the documentation I've seen says that to refresh this certificate I
need to put the host into maintenance mode and then re-enroll.. However I
cannot do that, because this is a single-host system so I cannot put the
host in local mode -- there is no place to migrate the VMs (let alone the
Engine VM).
So.... Is there a command-line way to re-enroll manually and update the
host certs? Or some other way to get all the leftover certs renewed?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of
the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and
obviously nothing in /etc/pki/ on the host itself.
--
Derek Atkins 617-623-3745
derek(a)ihtfp.com www.ihtfp.com
Computer and Internet Security Consultant
3 months, 3 weeks
