- Users - oVirt List Archives

oVirt 4.4. Engine Deployment: Problems with Gluster Storage Domain
by Thyen, Niko 18 Jul '23

18 Jul '23

Hi everybody! I am having a hard time getting oVirt 4.4 to work. We want to update our 4.3 Cluster and i am trying to set up a fresh 4.4 Cluster (and restore the backup later on) in order to update to 4.5. It fails at the end of the engine deloyment, when the Gluster Storage Domain should be added. I installed oVirt Node 4.4.10 on an old PC and made the following modifications to the engine deployment process: - altered defaults in /usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/defaults/main.yml: - "he_pause_before_engine_setup: true" (in this pause before engine setup, i ssh into the engine and exclude the package postgresql-jdbc from update, which otherwise breaks the deployment [1]) - "he_remove_appliance_rpm: false" (to avoid the large download every single try, i tried a lot) - "he_force_ip4: true" (to avoid problems with IPv6, see below) - in /usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/fetch_host_ip.yml i added after "- name: Get host address resolution:" the following lines (to avoid a problem with an "invalid" IPv6-Adress, which otherwise breaks the deployment [2]): - name: Get host IP addresses ansible.builtin.command: hostname -I register: hostname_addresses_output changed_when: true Most times, i started deployment via shell but tried via webinterface of the node as well. It fails at the task "Add glusterfs storage domain" with the following message: "[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400." (See also [3]) When the setup asks for storage, i tried different answers (gluster.local:/volume, gluster.local:/path/to/brick/volume, 192.168.8.51:/volume ...), no mount options. I added firewall rules for glusterfs at node and engine. Even tried disabling the firewall. No firewall on the gluster servers running. On the Node, i also tested setting SELinux to permissive. Recorded the traffic at different interfaces ("ovirtmgmt" and "virbr0" on the node and "eth0" on the engine) and i can see the node and the gluster server talking: Node gets the volume with options (which are, btw, compliant to the docs, "storage.owner-gid: 36" "storage.owner-uid: 36" etc) but thats it, no further packets to mount the volume. I noticed some ARP packets as well, the node asks the IP from the engine (the configured static IP, which is not yet active). And the engine sends a dns request for the gluster server to the node (via interface virbr0), but doesnt connect to the gluster server. At least, thats what i can see, most of the traffic is TLS, which i couldnt decrypt yet. I appreciate any hint where to find the right keys. Anyway, i can ssh from the engine to the gluster server and mount the gluster volume manually on the node (mount -t glusterfs gluster.local:/volume /local/path), so there seem no connectivity issues. Since the engine deployment log is around 30MB i attached a log summary with findings i found relevant. I'll provide more logs if needed. I really wanna put this huge timesink to an end. Can anyone help me or point me in the right direction? Many thanks in advance :) Regards, Niko [1] This was the error message i got: "[ ERROR ] fatal: [localhost -> 192.168.222.195]: FAILED! => {"attempts": 30, "changed": false, "connection": "close", "content": "Error500 - Internal Server Error", "content_encoding": "identity", "content_length": "86", "content_type": "text/html; charset=UTF-8", "date": "Wed, 17 May 2023 22:42:27 GMT", "elapsed": 0, "msg": "Status code was 500 and not [200]: HTTP Error 500: Internal Server Error", "redirected": false, "server": "Apache/2.4.37 (centos) OpenSSL/1.1.1k mod_auth_gssapi/1.6.1 mod_wsgi/4.6.4 Python/3.6", "status": 500, "url": "http://localhost/ovirt-engine/services/health"}" [2] This was the error message i got: "VDSM ovirt.martinwi.local command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Invalid IP address: 'fe80::ea3f:67ff:fe7f:a029%ovirtmgmt' does not appear to be an IPv4 or IPv6 address"}" [3] /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230718140628-rryscj.log: 2023-07-18 16:32:35,877+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:106 {'msg': 'Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400.', 'exception': 'Traceback (most recent call last):\n File "/tmp/ansible_ovirt_storage_domain_payload_b4ofbzxa/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py", line 804, in main\n File "/tmp/ansible_ovirt_storage_domain_payload_b4ofbzxa/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py", line 674, in create\n **kwargs\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py", line 26258, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n return future.wait() if wait else future\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 229, in callback\n self._check_fault(response)\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n self._raise_error(response, body)\n File "/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n raise error\novirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400.\n', 'invocation': {'module_args': {'state': 'unattached', 'name': 'hosted_storage', 'host': 'ovirt.martinwi.local', 'data_center': 'Default', 'wait': True, 'glusterfs': {'address': 'gluster1.martinwi.local', 'path': '/gv3', 'mount_options': ''}, 'timeout': 180, 'poll_interval': 3, 'fetch_nested': False, 'nested_attributes': [], 'domain_function': 'data', 'id': None, 'description': None, 'comment': None, 'localfs': None, 'nfs': None, 'iscsi': None, 'managed_block_storage': None, 'posixfs': None, 'fcp': None, 'wipe_after_delete': None, 'backup': None, 'critical_space_action_blocker': None, 'warning_low_space': None, 'destroy': None, 'format': None, 'discard_after_delete': None}}, '_ansible_no_log': False, 'changed': False}

1 0

engine-setup failing on 4.3.2 -> 4.3.3 fails during Engine schema refresh fail
by Edward Berger 18 Jul '23

18 Jul '23

I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings. ... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql [ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Attaching engine-setup logfile.

4 6

Local physical disk directly on the VM
by Jorge Visentini 17 Jul '23

17 Jul '23

Hi. Is it possible to deliver a physical disk straight to the VM? What's my idea... I have a local disk on the server and I want to deliver it directly to a VM, is it possible? For example, take the hypervisor disk /dev/disk/by-id/ata-INTEL_SSDSC2KB480G8_PHYF117203AA480BGN and hand it to the VM. BR. -- Att, Jorge Visentini +55 55 98432-9868

1 0

Re: Future of the oVirt CSI Driver
by Mike Rochefort 11 Jul '23

11 Jul '23

On 7/11/23 2:58 AM, Sandro Bonazzola wrote: > I think that if OKD is dropping the oVirt CSI Driver this also will mean > nobody will be actively developing it anymore. I asked about this on the Kubernetes Slack workspace and Vadim mentioned that in order for the OKD project to continue providing oVirt support, someone with oVirt knowledge and operator development would need to step up. It would also require forking the installer and a few other projects, which is something the OKD team spent a lot of effort to not have to do anymore during the 4.x series. https://github.com/openshift/installer Looking at the OpenShift Installer git, so far what's changed is an admin can no longer specify oVirt as a deployment target. According to Jira, there will be a second phase of the deprecation where the actual provisioning pieces (e.g. Terraform) will be removed, but that probably won't be for 4.14. https://issues.redhat.com//browse/OCPBUGS-14818 But not having anyone develop the CSI driver means it will likely end up with bit rot. With OpenShift 4.14 the oVirt platform becomes less attractive to use, though IPI/UPI installations should be possible. A different storage backend would probably be recommended, however. -- Mike Rochefort

2 1

Simple question - oVirt el8 with UEFI or el9 without UEFI?
by Jorge Visentini 11 Jul '23

11 Jul '23

We are in the process of migrating from version 4.4 to 4.5 and we are in doubt whether we use version el8 with UEFI or el9 without UEFI? What do you recommend? BR. -- Att, Jorge Visentini +55 55 98432-9868

1 0

Future of the oVirt CSI Driver
by Michael Rochefort 11 Jul '23

11 Jul '23

Starting with OpenShift / OKD 4.14, the oVirt CSI driver will no longer be delivered as part of the platform and the oVirt provisioning artifacts will not be included in the installer. Does this at all impact the development and maintenance plans for the driver itself? Original archived driver: https://github.com/oVirt/csi-driver OpenShift forked driver: https://github.com/openshift/ovirt-csi-driver Cheers, Mike

2 1

oVirt Self-Hosted Deploy in lopping during search available subnet
by Lucy Silvestre 11 Jul '23

11 Jul '23

Hi, I have been stuck for weeks in this problem during self-hosted deployment. Does someone have suggestions to resolve this? Error: The Deploy loops in this part until the Deployment Fail: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get ip route] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fail if can't find an available subnet] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set new IPv4 subnet prefix] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Search again with another prefix] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Define 3rd chunk] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set 3rd chunk] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get ip route] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fail if can't find an available subnet] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set new IPv4 subnet prefix] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Search again with another prefix] [ INFO ] ok: [localhost] Details: OS: Oracle Linux 8.8 (minimum installation) and updated oVirt 4.4 ovirt-hosted-engine-setup and ovirt-engine-appliance installed Network device used in the Host machine: eno1 - 192.168.10.x and I also tried with Bond. I made a reservation on DHCP for these IPs used during the installation. I tried by terminal and Cockpit, and it is the same issue. I tried with the command: hosted-engine --deploy --4 I tried with DHCP and Static IP I will appreciate every help. Thank you.

2 1

ACTION_TYPE_FAILED_DISK_IS_BEING_TRANSFERRED
by eshwayri＠gmail.com 08 Jul '23

08 Jul '23

System unexpectedly lost power. Now when I try to start one of the VMs I get: "ACTION_TYPE_FAILED_DISK_IS_BEING_TRANSFERRED". This may be due to a failed backup earlier in the week. There are no active tasks against this VM or its disk at this time. The disk shows OK in the storage view. Any ideas?

1 3

Suggestion to switch to nightly
by Sandro Bonazzola 07 Jul '23

07 Jul '23

Hi, As you probably noticed there were no regular releases after oVirt 4.5.4 <https://ovirt.org/release/4.5.4/> in December 2022. Despite the calls to action to the community and to the companies involved with oVirt, there have been no uptake of the leading of the oVirt project yet. The developers at Red Hat still dedicating time to the project are now facing the fact they lack the time to do formal releases despite they keep fixing platform regressions like the recent ones due to the new ansible changes. That makes a nightly snapshot setup a more stable environment than oVirt 4.5.4. For this reason, we would like to suggest the user community to enable nightly repositories for oVirt by following the procedure at: https://www.ovirt.org/develop/dev-process/install-nightly-snapshot.html This will ensure that the latest fixes for the platform regressions will be promptly available. Regards, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING - Red Hat In-Vehicle Operating System Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

5 9

GPU Passthrough issues with oVirt 4.5
by Vinícius Ferrão 07 Jul '23

07 Jul '23

Hello, does anyone is having issues with device passthrough on oVirt 4.5? I can passthrough the devices without issue to a given VM, but inside the VM it fails to recognize all the devices. In my case I’ve added 4x GPUs to a VM, but only one show up, and there’s the following errors inside the VM: [ 23.006655] nvidia 0000:0a:00.0: enabling device (0000 -> 0002) [ 23.008026] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR1 is 0M @ 0x0 (PCI:0000:0a:00.0) [ 23.008035] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR2 is 0M @ 0x0 (PCI:0000:0a:00.0) [ 23.008040] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR3 is 0M @ 0x0 (PCI:0000:0a:00.0) [ 23.008045] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR4 is 0M @ 0x0 (PCI:0000:0a:00.0) [ 23.008049] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR5 is 0M @ 0x0 (PCI:0000:0a:00.0) [ 23.012339] NVRM: The NVIDIA GPU 0000:0a:00.0 (PCI ID: 10de:1db1) NVRM: installed in this system is not supported by the NVRM: NVIDIA 535.54.03 driver release. NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products' NVRM: in this release's README, available on the operating system NVRM: specific graphics driver download page at www.nvidia.com. [ 23.016175] nvidia: probe of 0000:0a:00.0 failed with error -1 [ 23.016838] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR0 is 0M @ 0x0 (PCI:0000:0b:00.0) [ 23.016842] nvidia: probe of 0000:0b:00.0 failed with error -1 [ 23.017211] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: NVRM: BAR0 is 0M @ 0x0 (PCI:0000:0c:00.0) [ 23.017215] nvidia: probe of 0000:0c:00.0 failed with error -1 [ 23.017248] NVRM: The NVIDIA probe routine failed for 3 device(s). [ 23.214409] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 535.54.03 Tue Jun 6 22:20:39 UTC 2023 [ 23.485704] [drm] [nvidia-drm] [GPU ID 0x00000900] Loading driver [ 23.485708] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:09:00.0 on minor 1 On the host this shows up on dmesg, but seems right: [ 709.572845] vfio-pci 0000:1a:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 [ 709.572877] vfio-pci 0000:1a:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 [ 709.572883] vfio-pci 0000:1a:00.0: vfio_ecap_init: hiding ecap 0x23@0xac0 [ 710.660813] vfio-pci 0000:1d:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 [ 710.660845] vfio-pci 0000:1d:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 [ 710.660851] vfio-pci 0000:1d:00.0: vfio_ecap_init: hiding ecap 0x23@0xac0 [ 711.748760] vfio-pci 0000:1e:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 [ 711.748791] vfio-pci 0000:1e:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 [ 711.748797] vfio-pci 0000:1e:00.0: vfio_ecap_init: hiding ecap 0x23@0xac0 [ 712.836687] vfio-pci 0000:1c:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 [ 712.836718] vfio-pci 0000:1c:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 [ 712.836725] vfio-pci 0000:1c:00.0: vfio_ecap_init: hiding ecap 0x23@0xac0 Thanks.

2 1