November 2024 - Users - oVirt List Archives

Latest oVirt images
by Michael Thomas 11 Nov '24

11 Nov '24

tl;dr Where can I find a compatible set of host packages and hosted-engine image? What is the recommended combination to use for new installs? First, some background: I've been trying to get a new oVirt 4.5.5 install on Rocky 9 hosts using a hosted engine. My first few attempts failed because the engine image (ovirt-engine-appliance-4.5-20231201120201.1.el9) was still based on CentOS8-stream. Using --ansible-extra-vars=he_pause_before_engine_setup=true I was able to redirect the repos to vault.centos.org. This helped, but still failed when the engine tried to access the host: 2024-11-06 17:05:53,600-0600 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': 'Host is not up, please check logs, perhaps also on the engine machine', '_ansible_no_log': False, 'changed': False} 2024-11-06 17:05:53,700-0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} ...and the logs on the engine throw a NetworkNotFoundError while trying to set up OVN: "stdout" : "fatal: [hv1-mgmt.cds.ligo-la.caltech.edu] FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"10.110.115.21\", \"hv1-mgmt.cds.ligo-la.caltech.edu\"], \"delta\": \"0:00:02.413890\", \"end\": \"2024-11-08 14:29:54.215138\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-11-08 14:29:51.801248\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'hv1-mgmt.cds.ligo-la.caltech.edu'\\n\\nDuring handling of the above exception, another exception o ccurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: hv1-mgmt.cds.ligo-la.caltech.edu\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3. 9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\", \" r eturn networks[net_name]\", \"KeyError: 'hv1-mgmt.cds.ligo-la.caltech.edu'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: hv1-mgmt.cds.ligo-la.caltech.edu\"], \"stdout\": \"\", \"stdout_lines\": []}", Ok, so then I think to myself that I should be using a newer engine image. I installed ovirt-engine-appliance-4.5-20240817071039.1.el9.x86_64.rpm and tried again. But of course that failed because the host and engine now have incompatible versions: [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 154: Host hv1-mgmt.cds.ligo-la.caltech.edu is compatible with versions (4.2,4.3,4.4,4.5,4.6,4.7) and cannot join Cluster CDS which is set to version 4.8., code 1110: Host hv1-mgmt.cds.ligo-la.caltech.edu's following network(s) are not synchronized with their Logical Network configuration: ovirtmgmt., code 9000: Failed to verify Power Management configuration for Host hv1-mgmt.cds.ligo-la.caltech.edu., fix accordingly and re-deploy."} --Mike

2 4

Re: [External] : oVirt slow backups from ISCSI storage domain to nfs target
by Don Dupuis 11 Nov '24

11 Nov '24

Marcos Yes I understand what you are saying and that is great for a 1Gbps network, I have a 200Gbps network and I am only getting 140MiB/sec. There is just something different with how the data is being read out of this storage system. Don On Mon, Nov 11, 2024 at 9:40 AM Marcos Sungaila <marcos.sungaila(a)oracle.com> wrote: > Don, > > > > Sorry, I wasn’t clear in my previous e-mail. My comment about small disks > actually is about how much disk space you have currently in use, not the > virtual disk size. > > For example, I have a 200GB virtual disk VM with only 4,6 GB in use. It > takes about 40 sec to transfer the disk to an NFS share in my lab reaching > 128MiB/sec on a 1Gbps network connection. > > > > Marcos > > > > *From:* Don Dupuis <dondster(a)gmail.com> > *Sent:* Monday, November 11, 2024 11:32 AM > *To:* Marcos Sungaila <marcos.sungaila(a)oracle.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [External] : [ovirt-users] oVirt slow backups from ISCSI > storage domain to nfs target > > > > Marcos, > > These are not small files, the files I am talking about are about 400GB or > bigger. If I do the same thing on another DE6000 with SAS disks, I don't > see this issue. Thin provisioned disks are ok on both storage systems, But > the 6600F has NVME interfaced SSDs and runs this slow on preallocated > disks. For the short term we are just going to go back to thin provisioned > as we can't get the backups to finish in the allocated time windows. > > > > Don > > > > On Mon, Nov 11, 2024 at 5:23 AM Marcos Sungaila < > marcos.sungaila(a)oracle.com> wrote: > > Hi Don, > > > > When transferring small files to an NFS share, many times, they can fit > the in-memory cache, leading to high transfer rates. For bigger files, the > remote NFS storage will need to commit data to the disk once the cache is > filled. > > In general, it is common to see the initial data transfer at high numbers, > slowing down as the transfer continues. This behavior is not only with NFS > shares; you may face the same behavior using scp. > > Anyway, it is recommended that a transfer test be run using other methods > to confirm this is the cause. > > > > Marcos > > > > *From:* Don Dupuis <dondster(a)gmail.com> > *Sent:* Friday, November 8, 2024 5:06 PM > *To:* users <users(a)ovirt.org> > *Subject:* [External] : [ovirt-users] oVirt slow backups from ISCSI > storage domain to nfs target > > > > Hi > > I am using oVirt 4.4.10.7 with vms on iscsi Lenovo DE-6600F(nvme > interfaced ssds). Backing up thin vms performance is fine, but with > preallocated disks, backups to nfs storage with 200Gb interface is only > transferring qcow2 image to nfs at 140MiB per sec. Why is qemu-nbd so slow > with preallocated images? Does anyone have any issues related to this? Any > help would be appreciated. I have made iscsid.conf changes, linux kernel > boot changes, and still the same performance. > > > > Thanks > > Don > >

1 0

Re: [External] : oVirt slow backups from ISCSI storage domain to nfs target
by Don Dupuis 11 Nov '24

11 Nov '24

Marcos, These are not small files, the files I am talking about are about 400GB or bigger. If I do the same thing on another DE6000 with SAS disks, I don't see this issue. Thin provisioned disks are ok on both storage systems, But the 6600F has NVME interfaced SSDs and runs this slow on preallocated disks. For the short term we are just going to go back to thin provisioned as we can't get the backups to finish in the allocated time windows. Don On Mon, Nov 11, 2024 at 5:23 AM Marcos Sungaila <marcos.sungaila(a)oracle.com> wrote: > Hi Don, > > > > When transferring small files to an NFS share, many times, they can fit > the in-memory cache, leading to high transfer rates. For bigger files, the > remote NFS storage will need to commit data to the disk once the cache is > filled. > > In general, it is common to see the initial data transfer at high numbers, > slowing down as the transfer continues. This behavior is not only with NFS > shares; you may face the same behavior using scp. > > Anyway, it is recommended that a transfer test be run using other methods > to confirm this is the cause. > > > > Marcos > > > > *From:* Don Dupuis <dondster(a)gmail.com> > *Sent:* Friday, November 8, 2024 5:06 PM > *To:* users <users(a)ovirt.org> > *Subject:* [External] : [ovirt-users] oVirt slow backups from ISCSI > storage domain to nfs target > > > > Hi > > I am using oVirt 4.4.10.7 with vms on iscsi Lenovo DE-6600F(nvme > interfaced ssds). Backing up thin vms performance is fine, but with > preallocated disks, backups to nfs storage with 200Gb interface is only > transferring qcow2 image to nfs at 140MiB per sec. Why is qemu-nbd so slow > with preallocated images? Does anyone have any issues related to this? Any > help would be appreciated. I have made iscsid.conf changes, linux kernel > boot changes, and still the same performance. > > > > Thanks > > Don >

2 1

Re: [External] : open v-switch woes
by Marcos Sungaila 11 Nov '24

11 Nov '24

Rephrasing my last e-mail: Working with open vswitch at the ovirt infrastructure level is NOT an easy task. Marcos From: Marcos Sungaila via Users <users(a)ovirt.org> Sent: Monday, November 11, 2024 9:57 AM To: Tim Walsh <mr_tim_walsh(a)hotmail.com>; users(a)ovirt.org Subject: [ovirt-users] Re: [External] : open v-switch woes Tim, Working with open vswitch at the ovirt infrastructure level is an easy task. Here you have some steps to start with it. Deploying an OVN enabled system have the following requirements: * Stand-alone Engine installed on a bare metal or a VM outside the ovirt cluster. Running SHE is not possible since the default cluster will use Linux bridges, not openvswitch * No extra packages are required * After the Engine is deployed, create a new cluster, set the switch type to OVS and the default network provider to ovirt-provider-ovn * Add a host to the new cluster, note that migrating a host from a Linux bridge cluster to OVS may fail and may require you to clean-up all network configuration before joining the host to the right cluster. * In the network menu, you will need to create an ovn network for each tagged vlan, as well as for the untagged network you may have * If you enable network port security, you will need to create security groups and security rules before any communication can happen between instances, and to/from external ips Marcos From: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>> Sent: Friday, October 25, 2024 6:00 PM To: Marcos Sungaila <marcos.sungaila(a)oracle.com<mailto:marcos.sungaila@oracle.com>>; users(a)ovirt.org<mailto:users@ovirt.org> Subject: Re: [External] : [ovirt-users] open v-switch woes Yes, not only a desire to replicate the functionality of V-Switches on VM-Ware, but also to have an isolated test environment. Create a virtual firewall (using Opensense or pfSense or something) and create VMs behind the firewall that can all talk to each other, but the firewall only passes out the application traffic (for example a web app on Nginx, or Remote Desktop to the private Environment). Microsoft Hyper-V has "private" and "internal" switches that can be set up, (and thats the case on either a standalone or a cluster) so I figured oVirt must have something similar if not the same. these may or may not necessarily be tagged to a VLAN on a physical switch. Thanks, Tim ________________________________ From: Marcos Sungaila <marcos.sungaila(a)oracle.com<mailto:marcos.sungaila@oracle.com>> Sent: Thursday, October 24, 2024 4:22 PM To: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>; users(a)ovirt.org<mailto:users@ovirt.org> <users(a)ovirt.org<mailto:users@ovirt.org>> Subject: RE: [External] : [ovirt-users] open v-switch woes Hey Tim, Deploying an OVN-enabled cluster is not that trivial. There are many caveats to make it run. Is there any special use case you need to address to use OVS/OVN? Marcos From: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>> Sent: Thursday, October 24, 2024 12:28 AM To: users(a)ovirt.org<mailto:users@ovirt.org> Subject: [External] : [ovirt-users] open v-switch woes Hey community, I'm trying to get open v-switch set up to work like it does in VMware. I am running Rocky Linux 8.9 and oVirt 4.5.5 (el8) I got the repos but online feedback recommends installing openvswitch, and ovn-northd, ovn-central and ovn-host I got openvswitch installed, but the other three: ovn-northd, ovn-central and ovn-host seemt o be elusive even after adding the CentOS-Advanced-Virtualization.repo and uipdating all "CentOS-" repost to point to "vault" instead of "mirrorlist" Can someo0ne help me with what I'm missing? I've tried Bing CoPilot and ChatGPT LOL but they say add that repo and install those packages. Thanks, Tim

1 0

Re: [External] : open v-switch woes
by Marcos Sungaila 11 Nov '24

11 Nov '24

Tim, Working with open vswitch at the ovirt infrastructure level is an easy task. Here you have some steps to start with it. Deploying an OVN enabled system have the following requirements: * Stand-alone Engine installed on a bare metal or a VM outside the ovirt cluster. Running SHE is not possible since the default cluster will use Linux bridges, not openvswitch * No extra packages are required * After the Engine is deployed, create a new cluster, set the switch type to OVS and the default network provider to ovirt-provider-ovn * Add a host to the new cluster, note that migrating a host from a Linux bridge cluster to OVS may fail and may require you to clean-up all network configuration before joining the host to the right cluster. * In the network menu, you will need to create an ovn network for each tagged vlan, as well as for the untagged network you may have * If you enable network port security, you will need to create security groups and security rules before any communication can happen between instances, and to/from external ips Marcos From: Tim Walsh <mr_tim_walsh(a)hotmail.com> Sent: Friday, October 25, 2024 6:00 PM To: Marcos Sungaila <marcos.sungaila(a)oracle.com>; users(a)ovirt.org Subject: Re: [External] : [ovirt-users] open v-switch woes Yes, not only a desire to replicate the functionality of V-Switches on VM-Ware, but also to have an isolated test environment. Create a virtual firewall (using Opensense or pfSense or something) and create VMs behind the firewall that can all talk to each other, but the firewall only passes out the application traffic (for example a web app on Nginx, or Remote Desktop to the private Environment). Microsoft Hyper-V has "private" and "internal" switches that can be set up, (and thats the case on either a standalone or a cluster) so I figured oVirt must have something similar if not the same. these may or may not necessarily be tagged to a VLAN on a physical switch. Thanks, Tim ________________________________ From: Marcos Sungaila <marcos.sungaila(a)oracle.com<mailto:marcos.sungaila@oracle.com>> Sent: Thursday, October 24, 2024 4:22 PM To: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>; users(a)ovirt.org<mailto:users@ovirt.org> <users(a)ovirt.org<mailto:users@ovirt.org>> Subject: RE: [External] : [ovirt-users] open v-switch woes Hey Tim, Deploying an OVN-enabled cluster is not that trivial. There are many caveats to make it run. Is there any special use case you need to address to use OVS/OVN? Marcos From: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>> Sent: Thursday, October 24, 2024 12:28 AM To: users(a)ovirt.org<mailto:users@ovirt.org> Subject: [External] : [ovirt-users] open v-switch woes Hey community, I'm trying to get open v-switch set up to work like it does in VMware. I am running Rocky Linux 8.9 and oVirt 4.5.5 (el8) I got the repos but online feedback recommends installing openvswitch, and ovn-northd, ovn-central and ovn-host I got openvswitch installed, but the other three: ovn-northd, ovn-central and ovn-host seemt o be elusive even after adding the CentOS-Advanced-Virtualization.repo and uipdating all "CentOS-" repost to point to "vault" instead of "mirrorlist" Can someo0ne help me with what I'm missing? I've tried Bing CoPilot and ChatGPT LOL but they say add that repo and install those packages. Thanks, Tim

1 0

Re: [External] : Adding ISO doesn't work
by sol-pl-cm＠nttdata.com 08 Nov '24

08 Nov '24

Hi, Unfortunately, the proposed solution did not help. "Connection to ovirt-imageio service has failed. Ensure that ovirt-engine certificate is registered as a valid CA in the browser." and status "Paused by system" if I try to send the ISO anyway. Is there anything else I can check or improve?

2 1

Images upload - Content Security Policy
by jiri.mikulas＠mibcon.cz 08 Nov '24

08 Nov '24

Hello I'm using Oracle Linux with ovirt-engine 4.5.5-1.22.el8 Operating System: Oracle Linux Server 8.10 Kernel: Linux 5.15.0-301.163.5.2.el8uek.x86_64 After upgrading from ovirt-engine version 4.5.5-1.21.el8 to version 4.5.5-1.22.el8, there seems to be an issue with uploading images. When trying to test the connection for uploading images using Storage -> Domains -> MyStorage -> Disks -> Upload I debugged network communication in "chrome inspect" and received the error: "Refused to connect to https://ovirt-engine:54323/info/ because it appears in neither the connect-src directive nor the default-src directive of the Content Security Policy." Has anyone encountered this problem? I have the CA certificate properly imported since version 4.5.5-1.21.el8, where the uploads worked normally. Regards Jiri

2 1

Adding ISO doesn't work
by sol-pl-cm＠nttdata.com 08 Nov '24

08 Nov '24

I have a problem adding ISO. Fresh installation of OracleLinux-R8-U10 with Ovirt Version:4.5.5-1.22.el8 installed on it. I've already gone through all possible forums and threads with a similar problem, but none of them have been solved, and I haven't been able to solve the ISO problem either. Engine is installed as a virtual machine on VMware and manages two KVM hosts (also a fresh install). I replaced the certificates for the test, but it didn't help and I went back to the old configuration. I keep getting a certificate error when running the test, even though I added it to the browser. I tested on different browsers. "Connection to ovirt-imageio service has failed. Ensure that ovirt-engine certificate is registered as a valid CA in the browser." Can I get help here?

2 1

Need advice before building a new oVirt Environment
by gbigras＠telesat.com 08 Nov '24

08 Nov '24

My question will be down below but before that here is some background info: I currently have a functioning oVirt env (v4.5.7) with two self-hosted nodes (hand-me-down servers) and shared storage (TrueNAS server) with three data domains: one for the EngineVM (iSCSI), one for the VMs (iSCSI) and one for ISO files (NFS). The powers that be, want to move away from TrueNAS and hand-me-downs and have new servers and a new NAS appliance (most likely NetApp). They are the ones who did not want to spend the money on new gear but now they have changed their minds. I wish that I could have started my oVirt env on the new equipment. This new equipment will take at best 2 months to arrive. In the meantime, I have to create various VMs in my existing environment. Once my new gear arrives, what is the recommended approach: 1 - Start from scratch with a new datacenter/cluster env? Can I migrate VMs from one DC to another? The docs seem to say no. I'm guessing, I would have to export first, then import. 2 - Can my new equipment be part of the same DC but a different cluster? I was thinking that option #2 is the right approach. I could start with storage (ie: new NetApp) as part of the new cluster. Once that is part of the cluster, I can add hosts then finally migrate VMs. What say you? thank you in advance.

4 3

Ovirt Storage Monitoring behind RAID
by ankit＠eurus.net 05 Nov '24

05 Nov '24

Hi Folks, I have recently build Ovirt 4.5 cluster with server(DellXC 360 and 730xd) as storage using iscsi target where i want to monitor disks behing RAID(Perc Card). How can i monitor disks in ovirt itself ? Thanks. AS

2 1

Preparing to issue an oVirt release post Red Hat offboarding
by Sandro Bonazzola 01 Nov '24

01 Nov '24

Hi, Oracle fixed CVE-2024-7259 ( https://github.com/oVirt/ovirt-engine/security/advisories/GHSA-8xjm-j7fq-3w… ) in oVirt Engine and would be nice to issue an oVirt Engine release including the fix. As the current master also includes support for new cluster level 4.8, VDSM needs to be released as well providing cluster level 4.8. As you know, Red Hat offboarded the project and as such there are no more people officially assigned to the project from the company, despite a few of the historical developers still spend some of their free time helping out. I can help building the packages and getting them out on my free time but I have no capacity for doing any testing or documentation updates or release notes related to this release. Last oVirt Engine (4.5.6) was also released last February without any formal release announce. Any volunteer to look into differences between current master and current stable release and see if anything else needs to be released other than ovirt-engine and vdsm? Any volunteer to write release notes and update the website accordingly? Any volunteer to run ovirt-system-test on https://github.com/oVirt/ovirt-engine/pull/970 and https://github.com/oVirt/vdsm/pull/422 as OST doesn't run anymore on GitHub? Regards, -- Sandro Bonazzola

3 5