VM Migration Failed
by KSNull Zero
Running oVirt 4.4.5
VM cannot migrate between hosts.
vdsm.log contains the following error:
libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://ovhost01.local/system: authentication failed: Failed to verify peer's certificate
Certificates on hosts was renewed some time ago. How this issue can be fixed ?
Thank you.
2 weeks, 2 days
How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Derek Atkins
Hi,
I've got a single-host hosted-engine deployment that I originally
installed with 4.0 and have upgraded over the years to 4.3.10. I and some
of my users have upgraded remote-viewer and now I get an error when I try
to view the console of my VMs:
(remote-viewer:8252): Spice-WARNING **: 11:30:41.806:
../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error
in server certificate verification: CA signature digest algorithm too weak
(num=68:depth0:/O=<My Org Name>/CN=<Host's Name>)
I am 99.99% sure this is because the old certs use SHA1.
I reran engine-setup on the engine and it asked me if I wanted to renew
the PKI, and I answered yes. This replaced many[1] of the certificates in
/etc/pki/ovirt-engine/certs on the engine, but it did not update the
Host's certificate.
All the documentation I've seen says that to refresh this certificate I
need to put the host into maintenance mode and then re-enroll.. However I
cannot do that, because this is a single-host system so I cannot put the
host in local mode -- there is no place to migrate the VMs (let alone the
Engine VM).
So.... Is there a command-line way to re-enroll manually and update the
host certs? Or some other way to get all the leftover certs renewed?
Thanks,
-derek
[1] Not only did it not update the Host's cert, it did not update any of
the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and
obviously nothing in /etc/pki/ on the host itself.
--
Derek Atkins 617-623-3745
derek(a)ihtfp.com www.ihtfp.com
Computer and Internet Security Consultant
4 weeks
4.5.4 with Ceph only storage
by Maurice Burrows
Hey ... A long story short ... I have an existing Red Hat Virt / Gluster hyperconverged solution that I am moving away from.
I have an existing Ceph cluster that I primarily use for OpenStack and a small requirement for S3 via RGW.
I'm planning to build a new oVirt 4.5.4 cluster on RHEL9 using Ceph for all storage requirements. I've read many online articles on oVirt and Ceph, and they all seem to use the Ceph iSCSI gateway, which is now in maintenance, so I'm not real keen to commit to iSCSI.
So my question is, IS there any reason I cannot use CephFS for both hosted-engine and as a data storage domain?
I'm currently running Ceph Pacific FWIW.
Cheers
1 month
i can't access console with noVNC or VNC client(console.vv)
by z84614242＠163.com
i installed the ovirt 4.5 engine on centos stream 9 and add a ovirt node(ovirt node 4.5 iso) to this engine. i am going to run my vm on this node. i follow the instruction to create the data center, the cluster, the storage domain, upload the image. everything is fine. and after i create a vm with ubuntu image attach, i found that i can't visit the console. when i using the noVNC, it says "Something went wrong, connection is closed", when i visit vnc with virt-viewver, is says "Failed to complete handshake Error in the pull function". i try to change the console type to Bochs one and it appear the same. i change to QXL mode and the vm can't start any more. i check the log, it says "unsupported configuration: domain configuration does not support video model 'qxl'".
so now i can't visit my vm by anyway. i deploy the engine follow the official instruction and keep mostly option default but why still have this issue. why the noVNC says "Something went wrong" instead of telling me what is actually wrong
2 months, 1 week
Oracle Virtualization Manager 4.5 anyone?
by Thomas Hoberg
Redhat's decision to shut down RHV caught Oracle pretty unprepared, I'd guess, who had just shut down their own vSphere clone in favor of a RHV clone a couple of years ago.
Oracle is even less vocal about their "Oracle Virtualization" strategy, they don't even seem to have a proper naming convention or branding.
But they have been pushing out OV releases without a publicly announced EOL almost a year behind Redhat for the last years.
And after a 4.4 release in September 22, a few days ago on December 12th actually a release 4.5 was made public.
I've operated oVirt 4.3 with significant quality issues for some years and failed to make oVirt 4.4 work with any degree of acceptable stability but Oracle's variant of 4.4 proved to be rather better than 4.3 on CentOS7 with no noticable bugs, especially in the Hyperconverged setup that I am using with GlusterFS.
I assumed that this was because Oracle based their 4.4 in fact on RHV 4.4 and not oVirt, but since they're not telling, who knows?
One issue with 4.4 was that Oracle is pushing their UE-Kernel and that created immediate issues e.g. with VDO missing modules for UEK and other stuff, but that was solved easily enough by using the RHEL kernel.
With 4.5 Oracle obviously can't use RHV 4.5 as a base, because there is no such thing with RHV declared EOL and according to Oracle their 4.5 is based on oVirt 4.5.4, which made the quality of that release somewhat questionable, but perhaps they have spent the year that has passed since productively killing bugs... only to be caught by surprise again, I presume, by an oVirt release 4.5.5 on December 1st, that no one saw coming!
Long story slightly shorter, I've been testing Oracle's 4.5 variant a bit and it's not without issues.
But much worse, Oracle's variant of oVirt seems to be entirely without any community that I could find.
Now oVirt has been a somewhat secret society for years, but compared to what's going on with Oracle this forum is teaming with life!
So did I just not look around enough? Is there a secret lair where all those OV users are hiding?
Anyhow, here is what I've tested so far and where I'd love to have some feedback:
1. Setting up a three node HCI cluster from scratch using OL8.9 and OV 4.5
Since I don't have extra physical hardware for a 3 node HCI I'm using VMware workstation 17.5 on a Workstation running Windows 2022, a test platform that has been working for all kinds of virtualization tests from VMware ESXi, via Xcp-ng and ovirt.
Created three VMs with OL8.9 minimal and then installed OV 4.5. I used the UEK default kernels and then had an issue when Ansible is trying to create the (local) management engine: the VM simply could not reach the Oracle repo servers to install the packages inside the ME. Since that VM is entirely under the control of Ansible and no console access of any type is possible in that installation phase, I couldn't do diagnostics.
But with 4.4 I used to have similar issues and there switching back to the Redhat kernel for the ME (and the hosts) resolved them.
But with 4.5 it seems that UEK has become a baked-in dependency: the OV team doesn't even seem to do any testing with the Redhat kernel any more. Or not with the HCI setup, which has become deprecated somewhere in oVirt 4.4... Or not with the Cockpit wizard, which might be in a totally untested state, or....
Doing the same install on OL 8.9 with OV 4.4, however, did work just fine and I was even able to update to 4.5 afterwards, which was a nice surprise...
...that I could not repeat on my physical test farm using three Atoms. There switching to the UEK kernel on the hosts caused issues, hosts were becoming unresponsive, file systems inaccessible, even if they were perfectly fine at the Gluster CLI level and in the end the ME VM simply would not longer start. Switching back to the Redhat kernel resolved things there.
In short, switching between the Redhat kernel and UEK, which should be 100% transparent to all things userland including hypervisors, doesn't work.
But my attempts to go with a clean install of 4.5 on a Redhat kernel or UEK is also facing issues. So far the only thing that has worked was a single node HCI install using UEK and OV 4.5 and upgrading to OV 4.5 on a virtualized triple node OV 4.4 HCI cluster.
Anyone else out there trying these things?
I was mostly determined to move to Proxmox VE, but Oracle's OV 4.5 seemed to be handing a bit of a life-line to oVirt and the base architecture is just much more powerful (or less manual) than Proxmox, which doesn't have a management engine.
2 months, 4 weeks
Host deploy failure: Configure OVN for oVirt
by stephan.badenhorst＠fnb.co.za
Good day,
I am running into a problem during host deploy via the oVirt engine GUI since upgrading to ovirt-engine-4.5.5-1.el8. The "Configure OVN for oVirt" task seem to fail when trying to run the vdsm-tool ovn-config command. Host deploy used to work fine when the engine was on version 4.5.4.
Anyone that can guide me on the right path to get past this issue?
Does not seem to be a new problem - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/IDLGSBQFX35E...
Log extract:
2024-02-01 14:48:19 SAST - TASK [ovirt-provider-ovn-driver : Configure OVN for oVirt] *********************
.
.
.
"stdout" : "fatal: [mob-r1-l-ovirt-aa-1-23.x.fnb.co.za]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"192.168.2.100\", \"host23.mydomain.com\"], \"delta\": \"0:00:00.538143\", \"end
\": \"2024-02-01 14:48:20.596823\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-02-01 14:48:20.058680\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/t
ool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'host23.mydomain.com'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most rec
ent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n
ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn
_config.NetworkNotFoundError: host23.mydomain.com\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network
\", \" return networks[net_name]\", \"KeyError: 'host23.mydomain.com'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \
\\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address =
get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.
NetworkNotFoundError: host23.mydomain.com\"], \"stdout\": \"\", \"stdout_lines\": []}",
Thanks in advance!!
Stephan
3 months
Changing disk QoS causes segfault with IO-Threads enabled (oVirt 4.3.0.4-1.el7)
by jloh＠squiz.net
We recently upgraded to 4.3.0 and have found that when changing disk QoS settings on VMs whilst IO-Threads is enabled causes them to segfault and the VM to reboot. We've been able to replicate this across several VMs. VMs with IO-Threads disabled/turned off do not segfault when changing the QoS.
Mar 1 11:49:06 srvXX kernel: IO iothread1[30468]: segfault at fffffffffffffff8 ip 0000557649f2bd24 sp 00007f80de832f60 error 5 in qemu-kvm[5576498dd000+a03000]
Mar 1 11:49:06 srvXX abrt-hook-ccpp: invalid number 'iothread1'
Mar 1 11:49:11 srvXX libvirtd: 2019-03-01 00:49:11.116+0000: 13365: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer
Happy to supply some more logs to someone if they'll help but just wondering whether anyone else has experienced this or knows of a current fix other than turning io-threads off.
Cheers.
3 months
Keycloak SSO integration
by luc.lalonde＠polymtl.ca
Hello,
I've successfully added my AD LDAP service in Keycloak. My connection works and I am able to search for users from Keycloak.
How do I map an LDAP group 'adminusers' into the Ovirt web interface and map it to the internal administration group?
I just seem to see the ovirt-internal Keycloak authentication source in the base of my Ovirt URL.
Thank You!
3 months, 1 week
Can't create shared disk after upgrade to 4.5.4
by Igor Filipovic
Hi,
In my lab I'm using OLVM ( I know ... it's not original oVirt but it is based on it ). I was using 4.4.10, and after upgrade to 4.5.4 I'm no longer able to create shared disks, on any storage domain (nfs or fc). When I was on previous release I was able to create and use shared disks (required for cluster systems), but after upgrade when I try to create shared disk (preallocated) I'm getting an error saying : Error while executing action: Cannot add Virtual Disk. Disk's volume format is not supported for shareable disk.
In engine log I can see only this :
2024-03-31 20:04:28,997+02 WARN [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-37) [b8bc1831-260a-4253-b6fe-e0c81a6f81b6] Validation of action 'AddDisk' failed for user admin@internal-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__DISK,SHAREABLE_DISK_IS_NOT_SUPPORTED_BY_VOLUME_FORMAT
Does anyone have idea what could be the reason or how to solve this?
Thanks in advance and best regards
Igor
3 months, 2 weeks
Abnormal behavior on the reinstalled host
by ricardoot＠gmail.com
Hello everyone,
I need help again with a host that I have reinstalled l and I don't understand why it doesn't work well.
I have managed to get the host status to UP. But I can't migrate MV to this host. Yes, I can start a VM on this host and I can migrate it to another but it can't go back if you want.
These are the errors I see:
ID: 128
2024-03-19 20:34:56,809+01 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-77603) [6ea57120-4e64-4b1d-a31f- 1072f69c478b] EVENT_ID: VM_MIGRATION_TRYING_RERUN(128), Failed to migrate VM srvlinux to Host ovirt51.mydomain due to an Error: Fatal error during migration. Trying to migrate to another Host.
It also happens that I cannot access the noVNC console on this Host but the VM is running. It gives me this error:
"Something went wrong, connection is closed"
If I put the host in maintenance mode it does not activate again and gives an error:
Operation Canceled
Error while executing action:
ovirt51.mydomain:
Cannot activate Host. Host has no unique id.
Resolved by reinstalling the host and checking to activate it. But it doesn't seem normal to me.
Please does anyone know how to resolve these issues?
Thank you.
OS Version:
RHEL - 8.9 - 1.8.el8
OS Description:
Rocky Linux 8.9 (Green Obsidian)
Kernel Version:
4.18.0 - 513.18.1.el8_9.x86_64
KVM Version:
6.2.0 - 40.module+el8.9.0+1654+f4df84c4.2
LIBVIRT Version:
libvirt-8.0.0-22.module+el8.9.0+1405+b6048078
VDSM Version:
vdsm-4.50.6-3.git5d82b9e88.el8
SPICE Version:
0.14.3 - 4.el8
GlusterFS Version:
glusterfs-10.5-1.el8s
CEPH Version:
librbd1-16.2.15-1.el8s
Open vSwitch Version:
openvswitch-2.15-4.el8
Nmstate Version:
nmstate-1.4.5-2.el8_9
Kernel Features:
MDS: (Not affected), L1TF: (Not affected), SRBDS: (Not affected), MELTDOWN: (Not affected), RETBLEED: (Not affected), SPECTRE_V1: (Mitigation: usercopy/swapgs barriers and __user pointer sanitization), SPECTRE_V2: (Mitigation: Enhanced / Automatic IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: SW sequence), ITLB_MULTIHIT: (Not affected), MMIO_STALE_DATA: (Mitigation: Clear CPU buffers; SMT vulnerable), TSX_ASYNC_ABORT: (Not affected), SPEC_STORE_BYPASS: (Mitigation: Speculative Store Bypass disabled via prctl), GATHER_DATA_SAMPLING: (Mitigation: Microcode), SPEC_RSTACK_OVERFLOW: (Not affected)
VNC Encryption:
Enabled
OVN configured:
No
3 months, 3 weeks
