March 2024 - Users - oVirt List Archives

oVirt hosts mounting old storage domain even though it's not listed in the engine
by stanleym＠missouri.edu 19 Mar '24

19 Mar '24

We have an oVirt cluster with multiple hosts that continues to mount an old storage domain we used for ISO images. The engine does not show this old storage domain and I've even looked at the engine sql database and not found this old storage domain. Manually unmounting the ISO storage domain on the hosts doesn't fix it as it will remount within a few minutes. Is there some way to manually make vdsm stop remounting this old path? I didn't know if there is a way to regenerate those files from scratch. We've went through all our VMs and removed the cdroms even trying to get it to stop. We're going to try rebuilding a host from scratch to see if it's an artifact left on the host versus an artifact that's coming from the engine. Any help would be appreciated!

1 0

install node on centos stream 9 fails with message NetworkNotFoundError
by Jirka Simon 19 Mar '24

19 Mar '24

Hello oVirt community. i just tried to add a new host(wh refresh) to our cluster with installed Centos Stream 9, bud ovirt installation fails with message network not found. there are 4 NICs (2SPF one of them is configured) and 2 ethenrent which are not configured and are down here is end instalation log. any hint what can I check ? 2024-03-11 11:21:44 CET - { "uuid" : "c43fec42-2838-4319-8f00-adaf24351341", "counter" : 412, "stdout" : "fatal: [ovirt1.domain.tld]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"10.36.191.11\", \"ovirt1.domain.tld\"], \"delta\": \"0:00:02.236199\", \"end\": \"2024-03-11 11:21:43.909643\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-03-11 11:21:41.673444\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\n KeyError: 'ovirt1.domain.tld'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*arg s)\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 11 9, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_conf ig.py\\\", line 117, in get_network\", \" return networks[net_name]\", \"KeyError: 'ovirt1.domain.tld'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/us r/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_c aps(), net_name))\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\"], \"stdout\": \ "\", \"stdout_lines\": []}", "start_line" : 414, "end_line" : 415, "runner_ident" : "ca3d44fb-7482-481b-97d3-66f2ba7c9d0b", "event" : "runner_on_failed", "pid" : 1520750, "created" : "2024-03-11T10:21:43.926054", "parent_uuid" : "00163e21-72be-4886-1543-000000000042", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "4aa34ee9-1be3-4377-a573-ef33d837d406", "play" : "all", "play_uuid" : "00163e21-72be-4886-1543-000000000002", "play_pattern" : "all", "task" : "Configure OVN for oVirt", "task_uuid" : "00163e21-72be-4886-1543-000000000042", "task_action" : "ansible.builtin.command", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml:43", "role" : "ovirt-provider-ovn-driver", "host" : "ovirt1.domain.tld", "remote_addr" : "ovirt1.domain.tld", "res" : { "changed" : true, "stdout" : "", "stderr" : "Traceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network\n return networks[net_name]\nKeyError: 'ovirt1.domain.tld'\n\nDuring handling of the above exceptio n, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/bin/vdsm-tool\", line 195, in main\n return tool_command[cmd][\"command\"](*args)\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 63, in ov n_config\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network\n raise NetworkNotFoundError(net_name)\nvdsm.tool.ovn_config.NetworkNotFoundE rror: ovirt1.domain.tld", "rc" : 1, "cmd" : [ "vdsm-tool", "ovn-config", "10.36.191.11", "ovirt1.domain.tld" ], "start" : "2024-03-11 11:21:41.673444", "end" : "2024-03-11 11:21:43.909643", "delta" : "0:00:02.236199", "msg" : "non-zero return code", "invocation" : { "module_args" : { "_raw_params" : "vdsm-tool ovn-config 10.36.191.11 ovirt1.domain.tld\n", "_uses_shell" : false, "expand_argument_vars" : true, "stdin_add_newline" : true, "strip_empty_ends" : true, "argv" : null, "chdir" : null, "executable" : null, "creates" : null, "removes" : null, "stdin" : null } }, "stdout_lines" : [ ], "stderr_lines" : [ "Traceback (most recent call last):", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network", " return networks[net_name]", "KeyError: 'ovirt1.domain.tld'", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/usr/bin/vdsm-tool\", line 195, in main", " return tool_command[cmd][\"command\"](*args)", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn _config.py\", line 63, in ovn_config", " ip_address = get_ip_addr(get_network(network_caps(), net_name))", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network", " raise NetworkNotFoundError(net_name)", "v dsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld" ], "_ansible_no_log" : false }, "start" : "2024-03-11T10:21:41.493826", "end" : "2024-03-11T10:21:43.925759", "duration" : 2.431933, "ignore_errors" : null, "event_loop" : null, "uuid" : "c43fec42-2838-4319-8f00-adaf24351341" } } 2024-03-11 11:21:44 CET - { "uuid" : "c43fec42-2838-4319-8f00-adaf24351341", "counter" : 412, "stdout" : "fatal: [ovirt1.domain.tld]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"10.36.191.11\", \"ovirt1.domain.tld\"], \"delta\": \"0:00:02.236199\", \"end\": \"2024-03-11 11:21:43.909643\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-03-11 11:21:41.673444\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\n KeyError: 'ovirt1.domain.tld'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*arg s)\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 11 9, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_conf ig.py\\\", line 117, in get_network\", \" return networks[net_name]\", \"KeyError: 'ovirt1.domain.tld'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/us r/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_c aps(), net_name))\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\"], \"stdout\": \ "\", \"stdout_lines\": []}", "start_line" : 414, "end_line" : 415, "runner_ident" : "ca3d44fb-7482-481b-97d3-66f2ba7c9d0b", "event" : "runner_on_failed", "pid" : 1520750, "created" : "2024-03-11T10:21:43.926054", "parent_uuid" : "00163e21-72be-4886-1543-000000000042", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "4aa34ee9-1be3-4377-a573-ef33d837d406", "play" : "all", "play_uuid" : "00163e21-72be-4886-1543-000000000002", "play_pattern" : "all", "task" : "Configure OVN for oVirt", "task_uuid" : "00163e21-72be-4886-1543-000000000042", "task_action" : "ansible.builtin.command", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml:43", "role" : "ovirt-provider-ovn-driver", "host" : "ovirt1.domain.tld", "remote_addr" : "ovirt1.domain.tld", "res" : { "changed" : true, "stdout" : "", "stderr" : "Traceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network\n return networks[net_name]\nKeyError: 'ovirt1.domain.tld'\n\nDuring handling of the above exceptio n, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/bin/vdsm-tool\", line 195, in main\n return tool_command[cmd][\"command\"](*args)\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 63, in ov n_config\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network\n raise NetworkNotFoundError(net_name)\nvdsm.tool.ovn_config.NetworkNotFoundE rror: ovirt1.domain.tld", "rc" : 1, "cmd" : [ "vdsm-tool", "ovn-config", "10.36.191.11", "ovirt1.domain.tld" ], "start" : "2024-03-11 11:21:41.673444", "end" : "2024-03-11 11:21:43.909643", "delta" : "0:00:02.236199", "msg" : "non-zero return code", "invocation" : { "module_args" : { "_raw_params" : "vdsm-tool ovn-config 10.36.191.11 ovirt1.domain.tld\n", "_uses_shell" : false, "expand_argument_vars" : true, "stdin_add_newline" : true, "strip_empty_ends" : true, "argv" : null, "chdir" : null, "executable" : null, "creates" : null, "removes" : null, "stdin" : null } }, "stdout_lines" : [ ], "stderr_lines" : [ "Traceback (most recent call last):", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network", " return networks[net_name]", "KeyError: 'ovirt1.domain.tld'", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/usr/bin/vdsm-tool\", line 195, in main", " return tool_command[cmd][\"command\"](*args)", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn _config.py\", line 63, in ovn_config", " ip_address = get_ip_addr(get_network(network_caps(), net_name))", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network", " raise NetworkNotFoundError(net_name)", "v dsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld" ], "_ansible_no_log" : false }, "start" : "2024-03-11T10:21:41.493826", "end" : "2024-03-11T10:21:43.925759", "duration" : 2.431933, "ignore_errors" : null, "event_loop" : null, "uuid" : "c43fec42-2838-4319-8f00-adaf24351341" } } Thank you Jirka

2 4

Re: [External] : The table in the database is too large
by ziyi Liu 19 Mar '24

19 Mar '24

Is the DB vacuum cleaning thorough? I tried that a long time ago and it didn't seem to clear much space. Is the data recorded in DB cyclic or automatically cleared periodically? Is it necessary to enable global maintenance for DB vacuum?

1 0

Are there in oVirt any limitations on network interfaces?
by John Mortensen 15 Mar '24

15 Mar '24

Hi, We are experiencing some network limitations on our virtual machines, we have done some initial testing to pinpoint the exact issue. Hardware: 2 x 25G NICs per physical machine 2 x 100G interlinks between datacenters iPerf test between two VMs in same host on same cluster and same datacenter reveals ~20GB/s (what is more or less expected) iPerf test between two VMs in differet host on same cluster and same datacenter reveals ~20GB/s (what is more or less expected) iPerf test between two VMs in differet host on different cluster and different datacenter reveals ~10GB/s (what is half or less than expected) There are no firewalls or other identified blockeing equipment between the datacenters. The only difference, as I see it, are traffic going across the 100G interlinks. Are there anything that needs to be tuned / setup in the NICs either on the VM or the host to be able to get higher speed between the datacenters? /John

3 3

Renaming a datacenter and a cluster
by John Mortensen 15 Mar '24

15 Mar '24

Hi, I was wondering if someone has had the same challenge as I have. Due to new naming constraints we need to rename a few things, namely the following: 1. The "cluster" 2. The "data center" Looking through the documentation I have been unable to find this described and I am uncertain what consequences a rename can have on the environment. /John

1 0

delete old unuseful HostedEngine VM
by Nathanaël Blanchet 14 Mar '24

14 Mar '24

Hello I migrated back a self hosted engine to standalone vm, but I can't delete the old hosted engine vm and its related hosted storage domain. I tried to do it into the database (vm_static table) but I had this error: violates foreign key constraint "fk_snapshot_vm". How can I clean those two elements, the vm and the storage domain? -- Nathanaël Blanchet Administrateur Systèmes et Réseaux Service Informatique et REseau (SIRE) Département des systèmes d'information 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

1 0

The table in the database is too large
by ziyi Liu 14 Mar '24

14 Mar '24

numa_node 6.39GB vds_interface_statistics 3.08GB vm_statistics 2.64GB vds_statistics 1.72GB vm_interface_statistics 1.54GB audit_log 1.44GB disk_image_dynamic 1.22GB vm_dynamic 1GB These tables basically have more than 200 rows, how should I reduce the data? I have 111 hosts and 175 virtual machines

3 2

Default login Ovirt 4.5.5 node webconsole
by wim＠wibits.be 13 Mar '24

13 Mar '24

Hello, I have been trying to install Ovirt 4.5.5 for the very first time but I have been struggling. I install Ovirt using the documentation on the website: https://www.ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engin…. When I install my first node I am unable to access the webconsole on port 9090. I have set the root password and disabled the root lock. I am however unable to login on the webconsole with the root user. I have searched far and low but I am unable to find the correct information. Can someone tell me what the default login should be? Thank you, Wim

1 0

Storage Domain Limitations
by Erhan T 13 Mar '24

13 Mar '24

Hi, We have oVirt v4.4.9 environment on our organizitaion. There are 2TB storage domains which have one LUN each and the storage type is Fibre Channel. I would like to add 10TB or more sized LUNs and reduce the number of storage domain. Is it possible? Is there any limitation for storage domain size in this version of oVirt? I’m looking for documentation about this subject but I couldn't find any information. I would be so glad if you could help me.

2 1

Ovirt 4.5 HA over NFS fails when a single host goes down
by youssef.khristo＠elsewedyintel.com 12 Mar '24

12 Mar '24

Greetings, we have recently installed ovirt as a hosted-engine with high availability on six nodes over NFS storage (no Gluster), with power management through an on-board IPMI device, and the setup was successful. All the nodes (from Supermicro) are identical in every aspect, so no hardware differences exist and no modifications to the servers' hardware were performed. The hosted-engine was deployed on a second host, where two of the six hosts only were required to host the HE VM. The network interface on each node is bonded between two physical fiber optics NICs in LACP mode with a VLAN on top, serving as the sole networking interface for the server/node, no separate VM or storage networks were needed, as the host OS, hosted-engine vm, and storage are required to be on the same network and VLAN. We started by testing the high-availability of the hosted-engine VM (as it was deployed on two of the six nodes) by rebooting or powering off one of the hosts, and the VM would migrate successfully to the second HE node. The main goal of our experiments is to test the robustness of the setup, as it is required for the cluster to remain functional even when up to two hosts are brought down (whether due to a network or power issue), however, when rebooting or powering off one of the hosts, the HE VM goes down and takes the entire cluster with it, where we can't even access the web portal. Once the host is rebooted, the HE VM and the cluster becomes functional again. Sometimes the HE VM stays down for a set amount of time (5 to 6 minutes) and then goes back up, and sometimes it goes down until the problematic host is back up. This behavior happens to other VMs as well not the the HE. We suspected an issue with the NFS storage, however, during ovirt operation it is being mounted properly over /rhev/data-center/mnt/<nfs:directory>, while the expected behavior is for the cluster to stay operational and any other VMs to be migrated to other hosts. During one of the tests, we tried to mount the NFS storage on a different directory and there was no problem, we were even able to perform commands such as ls without any issues, as well as writing a text file at the directory's root, and be able to modify it normally. We suspected a couple of things the first being that the HE is unable to fence the problematic host (the one we took down), however, power management is setup properly. The other thing we suspected is the cluster hosts (after taking down one of them) are unable to acquire storage lease, which is weird since the host in question is down and non-operational, hence no locks should be in place. The reason behind this suspicion is the following two errors that we receive frequently when one host or more goes down from the engine\ovirt-engine\engine.log file: 1- "EVENT_ID: VM_DOWN_ERROR(119), VM HostedEngine is down with error. Exit message: resource busy: Failed to acquire lock: Lease is held by another host." 2- "[<id>] Command 'GetVmLeaseInfoVDSCommand( VmLeaseVDSParameters:{expectedEngineErrors='[NoSuchVmLeaseOnDomain]', storagePoolId='<pool-id>', ignoreFailoverLimit='false', leaseId='<lease-id>', storageDomainId='<domain-id>'})' execution failed: IRSGenericException: IRSErrorException: No such lease: 'lease=<lease-id>'" This is a third warning from the /var/log/vdsm/vdsm.log file 1- "WARN (check/loop) [storage.check] Checker '/rhev/data-center/mnt/<nfs-domain:/directory>/<id>/dom_md/metadata' is blocked for 310.00 seconds (check:265)" All the tests are done without setting nodes into maintenance mode as we are simulating an emergency situation. No HE configuration were modified via the config-engine command, the default values are used. Is this a normal behavior? Are we missing something? Do we need to tweak a certain configuration using the config-engine command to get a better behavior (e.g., shorter down period)? Best regards

5 6