Ceph-only storage for self-hosted engine
by Michael Thomas
I found this thread from last year that indicates it should be possible
to use ceph rbd for the self-hosted engine:
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/4VG4AWPQ2FBY...
In my new hosted-engine deployment, the hosted-engine --deploy menu only
shows me options for:
Please specify the storage you would like to use (glusterfs,
iscsi, fc, nfs)[nfs]:
Should I also see options for using ceph rbd, or do I need to start with
nfs, and migrate the self-hosted engine to rbd after the initial setup
is complete?
--Mike
1 month, 1 week
Latest oVirt images
by Michael Thomas
tl;dr Where can I find a compatible set of host packages and
hosted-engine image? What is the recommended combination to use for new
installs?
First, some background:
I've been trying to get a new oVirt 4.5.5 install on Rocky 9 hosts using
a hosted engine. My first few attempts failed because the engine image
(ovirt-engine-appliance-4.5-20231201120201.1.el9) was still based on
CentOS8-stream. Using
--ansible-extra-vars=he_pause_before_engine_setup=true I was able to
redirect the repos to vault.centos.org. This helped, but still failed
when the engine tried to access the host:
2024-11-06 17:05:53,600-0600 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:109 {'msg': 'Host is not up, please check
logs, perhaps also on the engine machine', '_ansible_no_log': False,
'changed': False}
2024-11-06 17:05:53,700-0600 ERROR
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:113 fatal: [localhost]: FAILED! =>
{"changed": false, "msg": "Host is not up, please check logs, perhaps
also on the engine machine"}
...and the logs on the engine throw a NetworkNotFoundError while trying
to set up OVN:
"stdout" : "fatal: [hv1-mgmt.cds.ligo-la.caltech.edu]: FAILED! =>
{\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\",
\"10.110.115.21\", \"hv1-mgmt.cds.ligo-la.caltech.edu\"], \"delta\":
\"0:00:02.413890\", \"end\": \"2024-11-08 14:29:54.215138\", \"msg\":
\"non-zero return code\", \"rc\": 1, \"start\": \"2024-11-08
14:29:51.801248\", \"stderr\": \"Traceback (most recent call last):\\n
File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\",
line 117, in get_network\\n return networks[net_name]\\nKeyError:
'hv1-mgmt.cds.ligo-la.caltech.edu'\\n\\nDuring handling of the above
exception, another exception o
ccurred:\\n\\nTraceback (most recent call last):\\n File
\\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return
tool_command[cmd][\\\"command\\\"](*args)\\n File
\\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line
63, in ovn_config\\n ip_address =
get_ip_addr(get_network(network_caps(), net_name))\\n File
\\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line
119, in get_network\\n raise
NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError:
hv1-mgmt.cds.ligo-la.caltech.edu\", \"stderr_lines\": [\"Traceback (most
recent call last):\", \" File \\\"/usr/lib/python3.
9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\",
\" r
eturn networks[net_name]\", \"KeyError:
'hv1-mgmt.cds.ligo-la.caltech.edu'\", \"\", \"During handling of the
above exception, another exception occurred:\", \"\", \"Traceback (most
recent call last):\", \" File \\\"/usr/bin/vdsm-tool\\\", line 195, in
main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \"
File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\",
line 63, in ovn_config\", \" ip_address =
get_ip_addr(get_network(network_caps(), net_name))\", \" File
\\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line
119, in get_network\", \" raise NetworkNotFoundError(net_name)\",
\"vdsm.tool.ovn_config.NetworkNotFoundError:
hv1-mgmt.cds.ligo-la.caltech.edu\"], \"stdout\": \"\", \"stdout_lines\":
[]}",
Ok, so then I think to myself that I should be using a newer engine
image. I installed
ovirt-engine-appliance-4.5-20240817071039.1.el9.x86_64.rpm and tried
again. But of course that failed because the host and engine now have
incompatible versions:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The
host has
been set in non_operational status, deployment errors: code 154:
Host hv1-mgmt.cds.ligo-la.caltech.edu is compatible with versions
(4.2,4.3,4.4,4.5,4.6,4.7) and cannot join Cluster CDS which is set to
version 4.8., code 1110: Host hv1-mgmt.cds.ligo-la.caltech.edu's
following network(s) are not synchronized with their Logical Network
configuration: ovirtmgmt., code 9000: Failed to verify Power
Management configuration for Host hv1-mgmt.cds.ligo-la.caltech.edu.,
fix accordingly and re-deploy."}
--Mike
1 month, 1 week
Re: [External] : oVirt slow backups from ISCSI storage domain to nfs target
by Don Dupuis
Marcos
Yes I understand what you are saying and that is great for a 1Gbps network,
I have a 200Gbps network and I am only getting 140MiB/sec. There is just
something different with how the data is being read out of this storage
system.
Don
On Mon, Nov 11, 2024 at 9:40 AM Marcos Sungaila <marcos.sungaila(a)oracle.com>
wrote:
> Don,
>
>
>
> Sorry, I wasn’t clear in my previous e-mail. My comment about small disks
> actually is about how much disk space you have currently in use, not the
> virtual disk size.
>
> For example, I have a 200GB virtual disk VM with only 4,6 GB in use. It
> takes about 40 sec to transfer the disk to an NFS share in my lab reaching
> 128MiB/sec on a 1Gbps network connection.
>
>
>
> Marcos
>
>
>
> *From:* Don Dupuis <dondster(a)gmail.com>
> *Sent:* Monday, November 11, 2024 11:32 AM
> *To:* Marcos Sungaila <marcos.sungaila(a)oracle.com>
> *Cc:* users <users(a)ovirt.org>
> *Subject:* Re: [External] : [ovirt-users] oVirt slow backups from ISCSI
> storage domain to nfs target
>
>
>
> Marcos,
>
> These are not small files, the files I am talking about are about 400GB or
> bigger. If I do the same thing on another DE6000 with SAS disks, I don't
> see this issue. Thin provisioned disks are ok on both storage systems, But
> the 6600F has NVME interfaced SSDs and runs this slow on preallocated
> disks. For the short term we are just going to go back to thin provisioned
> as we can't get the backups to finish in the allocated time windows.
>
>
>
> Don
>
>
>
> On Mon, Nov 11, 2024 at 5:23 AM Marcos Sungaila <
> marcos.sungaila(a)oracle.com> wrote:
>
> Hi Don,
>
>
>
> When transferring small files to an NFS share, many times, they can fit
> the in-memory cache, leading to high transfer rates. For bigger files, the
> remote NFS storage will need to commit data to the disk once the cache is
> filled.
>
> In general, it is common to see the initial data transfer at high numbers,
> slowing down as the transfer continues. This behavior is not only with NFS
> shares; you may face the same behavior using scp.
>
> Anyway, it is recommended that a transfer test be run using other methods
> to confirm this is the cause.
>
>
>
> Marcos
>
>
>
> *From:* Don Dupuis <dondster(a)gmail.com>
> *Sent:* Friday, November 8, 2024 5:06 PM
> *To:* users <users(a)ovirt.org>
> *Subject:* [External] : [ovirt-users] oVirt slow backups from ISCSI
> storage domain to nfs target
>
>
>
> Hi
>
> I am using oVirt 4.4.10.7 with vms on iscsi Lenovo DE-6600F(nvme
> interfaced ssds). Backing up thin vms performance is fine, but with
> preallocated disks, backups to nfs storage with 200Gb interface is only
> transferring qcow2 image to nfs at 140MiB per sec. Why is qemu-nbd so slow
> with preallocated images? Does anyone have any issues related to this? Any
> help would be appreciated. I have made iscsid.conf changes, linux kernel
> boot changes, and still the same performance.
>
>
>
> Thanks
>
> Don
>
>
1 month, 1 week
Re: [External] : oVirt slow backups from ISCSI storage domain to nfs target
by Don Dupuis
Marcos,
These are not small files, the files I am talking about are about 400GB or
bigger. If I do the same thing on another DE6000 with SAS disks, I don't
see this issue. Thin provisioned disks are ok on both storage systems, But
the 6600F has NVME interfaced SSDs and runs this slow on preallocated
disks. For the short term we are just going to go back to thin provisioned
as we can't get the backups to finish in the allocated time windows.
Don
On Mon, Nov 11, 2024 at 5:23 AM Marcos Sungaila <marcos.sungaila(a)oracle.com>
wrote:
> Hi Don,
>
>
>
> When transferring small files to an NFS share, many times, they can fit
> the in-memory cache, leading to high transfer rates. For bigger files, the
> remote NFS storage will need to commit data to the disk once the cache is
> filled.
>
> In general, it is common to see the initial data transfer at high numbers,
> slowing down as the transfer continues. This behavior is not only with NFS
> shares; you may face the same behavior using scp.
>
> Anyway, it is recommended that a transfer test be run using other methods
> to confirm this is the cause.
>
>
>
> Marcos
>
>
>
> *From:* Don Dupuis <dondster(a)gmail.com>
> *Sent:* Friday, November 8, 2024 5:06 PM
> *To:* users <users(a)ovirt.org>
> *Subject:* [External] : [ovirt-users] oVirt slow backups from ISCSI
> storage domain to nfs target
>
>
>
> Hi
>
> I am using oVirt 4.4.10.7 with vms on iscsi Lenovo DE-6600F(nvme
> interfaced ssds). Backing up thin vms performance is fine, but with
> preallocated disks, backups to nfs storage with 200Gb interface is only
> transferring qcow2 image to nfs at 140MiB per sec. Why is qemu-nbd so slow
> with preallocated images? Does anyone have any issues related to this? Any
> help would be appreciated. I have made iscsid.conf changes, linux kernel
> boot changes, and still the same performance.
>
>
>
> Thanks
>
> Don
>
1 month, 1 week
Re: [External] : open v-switch woes
by Marcos Sungaila
Rephrasing my last e-mail:
Working with open vswitch at the ovirt infrastructure level is NOT an easy task.
Marcos
From: Marcos Sungaila via Users <users(a)ovirt.org>
Sent: Monday, November 11, 2024 9:57 AM
To: Tim Walsh <mr_tim_walsh(a)hotmail.com>; users(a)ovirt.org
Subject: [ovirt-users] Re: [External] : open v-switch woes
Tim,
Working with open vswitch at the ovirt infrastructure level is an easy task.
Here you have some steps to start with it.
Deploying an OVN enabled system have the following requirements:
* Stand-alone Engine installed on a bare metal or a VM outside the ovirt cluster. Running SHE is not possible since the default cluster will use Linux bridges, not openvswitch
* No extra packages are required
* After the Engine is deployed, create a new cluster, set the switch type to OVS and the default network provider to ovirt-provider-ovn
* Add a host to the new cluster, note that migrating a host from a Linux bridge cluster to OVS may fail and may require you to clean-up all network configuration before joining the host to the right cluster.
* In the network menu, you will need to create an ovn network for each tagged vlan, as well as for the untagged network you may have
* If you enable network port security, you will need to create security groups and security rules before any communication can happen between instances, and to/from external ips
Marcos
From: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>
Sent: Friday, October 25, 2024 6:00 PM
To: Marcos Sungaila <marcos.sungaila(a)oracle.com<mailto:marcos.sungaila@oracle.com>>; users(a)ovirt.org<mailto:users@ovirt.org>
Subject: Re: [External] : [ovirt-users] open v-switch woes
Yes, not only a desire to replicate the functionality of V-Switches on VM-Ware, but also to have an isolated test environment.
Create a virtual firewall (using Opensense or pfSense or something) and create VMs behind the firewall that can all talk to each other, but the firewall only passes out the application traffic (for example a web app on Nginx, or Remote Desktop to the private Environment).
Microsoft Hyper-V has "private" and "internal" switches that can be set up, (and thats the case on either a standalone or a cluster) so I figured oVirt must have something similar if not the same. these may or may not necessarily be tagged to a VLAN on a physical switch.
Thanks,
Tim
________________________________
From: Marcos Sungaila <marcos.sungaila(a)oracle.com<mailto:marcos.sungaila@oracle.com>>
Sent: Thursday, October 24, 2024 4:22 PM
To: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>; users(a)ovirt.org<mailto:users@ovirt.org> <users(a)ovirt.org<mailto:users@ovirt.org>>
Subject: RE: [External] : [ovirt-users] open v-switch woes
Hey Tim,
Deploying an OVN-enabled cluster is not that trivial.
There are many caveats to make it run.
Is there any special use case you need to address to use OVS/OVN?
Marcos
From: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>
Sent: Thursday, October 24, 2024 12:28 AM
To: users(a)ovirt.org<mailto:users@ovirt.org>
Subject: [External] : [ovirt-users] open v-switch woes
Hey community,
I'm trying to get open v-switch set up to work like it does in VMware. I am running Rocky Linux 8.9 and oVirt 4.5.5 (el8)
I got the repos but online feedback recommends installing openvswitch, and ovn-northd, ovn-central and ovn-host
I got openvswitch installed, but the other three: ovn-northd, ovn-central and ovn-host seemt o be elusive even after adding the CentOS-Advanced-Virtualization.repo and uipdating all "CentOS-" repost to point to "vault" instead of "mirrorlist"
Can someo0ne help me with what I'm missing? I've tried Bing CoPilot and ChatGPT LOL but they say add that repo and install those packages.
Thanks,
Tim
1 month, 1 week
Re: [External] : open v-switch woes
by Marcos Sungaila
Tim,
Working with open vswitch at the ovirt infrastructure level is an easy task.
Here you have some steps to start with it.
Deploying an OVN enabled system have the following requirements:
* Stand-alone Engine installed on a bare metal or a VM outside the ovirt cluster. Running SHE is not possible since the default cluster will use Linux bridges, not openvswitch
* No extra packages are required
* After the Engine is deployed, create a new cluster, set the switch type to OVS and the default network provider to ovirt-provider-ovn
* Add a host to the new cluster, note that migrating a host from a Linux bridge cluster to OVS may fail and may require you to clean-up all network configuration before joining the host to the right cluster.
* In the network menu, you will need to create an ovn network for each tagged vlan, as well as for the untagged network you may have
* If you enable network port security, you will need to create security groups and security rules before any communication can happen between instances, and to/from external ips
Marcos
From: Tim Walsh <mr_tim_walsh(a)hotmail.com>
Sent: Friday, October 25, 2024 6:00 PM
To: Marcos Sungaila <marcos.sungaila(a)oracle.com>; users(a)ovirt.org
Subject: Re: [External] : [ovirt-users] open v-switch woes
Yes, not only a desire to replicate the functionality of V-Switches on VM-Ware, but also to have an isolated test environment.
Create a virtual firewall (using Opensense or pfSense or something) and create VMs behind the firewall that can all talk to each other, but the firewall only passes out the application traffic (for example a web app on Nginx, or Remote Desktop to the private Environment).
Microsoft Hyper-V has "private" and "internal" switches that can be set up, (and thats the case on either a standalone or a cluster) so I figured oVirt must have something similar if not the same. these may or may not necessarily be tagged to a VLAN on a physical switch.
Thanks,
Tim
________________________________
From: Marcos Sungaila <marcos.sungaila(a)oracle.com<mailto:marcos.sungaila@oracle.com>>
Sent: Thursday, October 24, 2024 4:22 PM
To: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>; users(a)ovirt.org<mailto:users@ovirt.org> <users(a)ovirt.org<mailto:users@ovirt.org>>
Subject: RE: [External] : [ovirt-users] open v-switch woes
Hey Tim,
Deploying an OVN-enabled cluster is not that trivial.
There are many caveats to make it run.
Is there any special use case you need to address to use OVS/OVN?
Marcos
From: Tim Walsh <mr_tim_walsh(a)hotmail.com<mailto:mr_tim_walsh@hotmail.com>>
Sent: Thursday, October 24, 2024 12:28 AM
To: users(a)ovirt.org<mailto:users@ovirt.org>
Subject: [External] : [ovirt-users] open v-switch woes
Hey community,
I'm trying to get open v-switch set up to work like it does in VMware. I am running Rocky Linux 8.9 and oVirt 4.5.5 (el8)
I got the repos but online feedback recommends installing openvswitch, and ovn-northd, ovn-central and ovn-host
I got openvswitch installed, but the other three: ovn-northd, ovn-central and ovn-host seemt o be elusive even after adding the CentOS-Advanced-Virtualization.repo and uipdating all "CentOS-" repost to point to "vault" instead of "mirrorlist"
Can someo0ne help me with what I'm missing? I've tried Bing CoPilot and ChatGPT LOL but they say add that repo and install those packages.
Thanks,
Tim
1 month, 1 week
Re: [External] : Adding ISO doesn't work
by sol-pl-cm＠nttdata.com
Hi,
Unfortunately, the proposed solution did not help.
"Connection to ovirt-imageio service has failed. Ensure that ovirt-engine certificate is registered as a valid CA in the
browser." and status "Paused by system" if I try to send the ISO anyway.
Is there anything else I can check or improve?
1 month, 1 week
Images upload - Content Security Policy
by jiri.mikulas＠mibcon.cz
Hello
I'm using Oracle Linux with ovirt-engine 4.5.5-1.22.el8
Operating System: Oracle Linux Server 8.10
Kernel: Linux 5.15.0-301.163.5.2.el8uek.x86_64
After upgrading from ovirt-engine version 4.5.5-1.21.el8 to version 4.5.5-1.22.el8, there seems to be an issue with uploading images.
When trying to test the connection for uploading images using
Storage -> Domains -> MyStorage -> Disks -> Upload
I debugged network communication in "chrome inspect" and received the error:
"Refused to connect to https://ovirt-engine:54323/info/ because it appears in neither the connect-src directive nor the default-src directive of the Content Security Policy."
Has anyone encountered this problem?
I have the CA certificate properly imported since version 4.5.5-1.21.el8, where the uploads worked normally.
Regards
Jiri
1 month, 1 week
Adding ISO doesn't work
by sol-pl-cm＠nttdata.com
I have a problem adding ISO. Fresh installation of OracleLinux-R8-U10 with Ovirt Version:4.5.5-1.22.el8 installed on it. I've already gone through all possible forums and threads with a similar problem, but none of them have been solved, and I haven't been able to solve the ISO problem either.
Engine is installed as a virtual machine on VMware and manages two KVM hosts (also a fresh install). I replaced the certificates for the test, but it didn't help and I went back to the old configuration.
I keep getting a certificate error when running the test, even though I added it to the browser. I tested on different browsers. "Connection to ovirt-imageio service has failed. Ensure that ovirt-engine certificate is registered as a valid CA in the browser."
Can I get help here?
1 month, 1 week
Need advice before building a new oVirt Environment
by gbigras＠telesat.com
My question will be down below but before that here is some background info:
I currently have a functioning oVirt env (v4.5.7) with two self-hosted nodes (hand-me-down servers) and shared storage (TrueNAS server) with three data domains: one for the EngineVM (iSCSI), one for the VMs (iSCSI) and one for ISO files (NFS).
The powers that be, want to move away from TrueNAS and hand-me-downs and have new servers and a new NAS appliance (most likely NetApp). They are the ones who did not want to spend the money on new gear but now they have changed their minds. I wish that I could have started my oVirt env on the new equipment. This new equipment will take at best 2 months to arrive. In the meantime, I have to create various VMs in my existing environment.
Once my new gear arrives, what is the recommended approach:
1 - Start from scratch with a new datacenter/cluster env? Can I migrate VMs from one DC to another? The docs seem to say no. I'm guessing, I would have to export first, then import.
2 - Can my new equipment be part of the same DC but a different cluster?
I was thinking that option #2 is the right approach. I could start with storage (ie: new NetApp) as part of the new cluster. Once that is part of the cluster, I can add hosts then finally migrate VMs.
What say you?
thank you in advance.
1 month, 1 week
