- Users - oVirt List Archives

stomp timeout in org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient
by Chris Smith 08 May '24

08 May '24

Hi all, I'm randomly getting this in the engine logs; These times are UTC+1 so are different to the ovirt host logs. 2024-05-06 21:10:47,238+01 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host '10.10.3.36', last response arrived 1604 ms ago. 2024-05-06 21:10:51,750+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to /10.10.3.36 2024-05-06 21:11:18,994+01 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host '10.10.3.36', last response arrived 1885 ms ago. 2024-05-06 21:11:22,197+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to /10.10.3.36 2024-05-06 21:11:27,767+01 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host '10.10.3.36', last response arrived 1779 ms ago. and I'm trying to debug why it can't connect. The vdsm logs on the ovirt host don't give me many clues about what's happening. I see the stomp requests; These times are UTC; 2024-05-06 20:10:51,807+0000 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff: 10.10.3.125:36296 (protocoldetector:125) 2024-05-06 20:10:51,807+0000 INFO (Reactor thread) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2024-05-06 20:10:51,808+0000 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2024-05-06 20:10:51,808+0000 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2024-05-06 20:11:23,240+0000 INFO (Reactor thread) [ProtocolDetector.Detector] Detected protocol stomp from ::ffff: 10.10.3.125:36302 (protocoldetector:125) 2024-05-06 20:11:23,416+0000 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Processing CONNECT request (stompserver:95) 2024-05-06 20:11:23,505+0000 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) 2024-05-06 20:11:23,505+0000 INFO (JsonRpc (StompReactor)) [Broker.StompAdapter] Subscribe command received (stompserver:124) but there's nothing around these entries to tell me what's causing the timeout. Is there a script or command I can run to simulate what's happening here to try and figure out what's going on? Thanks, -- Chris Smith Hosting Systems Administrator P +61 2 9045 2800 E csmith(a)squiz.net Level 1, 435a Kent Street, Sydney NSW 2000 www.squiz.net -- Squiz named leader GigaOm's Digital Experience Platform Radar report, 2023-24. Read about the report <https://www.squiz.net/blog/squiz-dxp-platform-leader-in-gigaom-radar-report>!

1 0

oVirt 4.5.6 + Windows 11 + TPM oddness
by Daniel Henninger 07 May '24

07 May '24

Hi folk, I'm running into a very bizarre situation. To start with, I am able to install Windows 11 perfectly fine, with or without Secure Boot enabled, and the TPM appears to function regardless. From the Windows OS itself, I can confirm that Get-TPM, tpm.msc, windows security, etc all show that the TPM looks happy and healthy and well received. However -- the system logs are absolutely flooded with the following error: The initialization of the Trusted Platform Module (TPM) failed. The TPM may be in failure mode. To allow diagnosis, contact the TPM manufacturer with the attached information. And this appears to correlate with SCCM being unable to properly set up a client certificate and register. But I am not exagerating when I say it's flooding the logs -- I am currently staring at a system log with 668 log entries, 97% of which are the same TPM error repeated. Does anyone have any ideas what might be going on or have seen this before? I'm running out of ideas. =/ Thanks!

1 0

equivalent to VMrare RDM
by Nathanaël Blanchet 06 May '24

06 May '24

Hello, I'd like to use dell csi driver for dynamic provisionning into vms like described here: https://dell.github.io/csm-docs/docs/deployment/csmoperator/drivers/powerma… Does something equivalent to RDM (Raw Device Mapping) exist with oVirt? https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.storage.do… I'm aware about host devices features, but tht is not relevant in this case. Thank you. -- Nathanaël Blanchet Administrateur Systèmes et Réseaux Service Informatique et REseau (SIRE) Département des systèmes d'information 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

1 0

Cannot create bond
by dcaamano＠apexutilities.ca 06 May '24

06 May '24

Hello, I recently deploy two KVM hosts and one KVM engine. things seem to be working good so far; however, I cannot create bonds in my KVM hosts. I have 4 interfaces, two enabled (cable connected) and two disabled (cable unplugged, not needed at the moment). The ovirmgmt network is running on one of the active interfaces. I have done the following: Create LACP bond with both connected interfaces. Create Load-balancing XOR bond with both connected interfaces. Create LACP bond with both unplugged interfaces. Create Load-balancing XOR bond with both unplugged interfaces. Create LACP bond with one connected and one unplugged interface. Create Load-balancing XOR bond with one connected and one interfaces. Remove server from KVM engine, create bond at the host level using cockpit, try to add the host back into KVM cluster. None of the options i have tried successfully create the bond in the KVM cluster. I will appreciate your assistance in getting this to work. Environment details OS on all servers: Oracle Linux 8.9 with kernel 5.15.0-205.149.5.1.el8uek.x86_64 KVM engine: running on a virtual machine in vCenter KVM hosts: physical servers Software version: 4.5.4-1.0.31.el8 Thank you

2 3

ARM64 Host Failed to Add to Cluster
by limajasond＠yahoo.com 06 May '24

06 May '24

Hi, I configured a Rocky Linux 9 ARM64 Raspberry Pi 4B as an oVirt Host. When adding to a new Cluster under an established Datacenter it fails configuring OVN, I did make sure ovirt-provider-ovn and ovirt-provider-ovn-driver was installed. The error I found after digging into the ansible yml file on the Hosted Engine and the Python py related to configuring OVN. Ultimately the error is below: [root@ovirtnode3 ~]# vdsm-tool ovn-config 192.168.1.50 ovirtnode3.test.com Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py", line 117, in get_network return networks[net_name] KeyError: 'ovirtnode3.test.com' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 195, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py", line 63, in ovn_config ip_address = get_ip_addr(get_network(network_caps(), net_name)) File "/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py", line 119, in get_network raise NetworkNotFoundError(net_name) vdsm.tool.ovn_config.NetworkNotFoundError: ovirtnode3.test.com Running vdsm-tool list-nets displays nothing. Something is off in the script, but I admit I am not sure how. I know oVirt says Aarch64 is experimental, but this is unusable if vdsm-tool can't even gather/list the networks or config the OVN.

2 2

Re: Ovirt Hyperconverged Storage
by Thomas Hoberg 29 Apr '24

29 Apr '24

And I might have misread where your problems actually are... Because oVirt was born on SAN but tries to be storage agnostic, it creates its own overlay abstraction, a block layer that is then managed within oVirt even when you use NFS or GlusterFS underneath. "The ISO domain" has actually been deprecated and ISO images can be put into any domain type (e.g. also data). But they still have to be uploaded to that domain via the management engine GUI, you can't just copy the ISO images somewhere within the files and directories oVirt might create and expect them to be visible to the GUI or the VMs.

1 0

Re: Ovirt Hyperconverged Storage
by Thomas Hoberg 29 Apr '24

29 Apr '24

Hi Tim, HA, HCI and failover either require or at least benefit from consistent storage. The original NFS reduce the risk of inconsistency to single files, Gluster puts the onus of consistency mostly the clients and I guess Ceph is similar. iSCSI has been described as a bit the worst of everything in storage and I can appreciate that view in a HA scenario because it doesn't help with consistency. Of course, its block layer abstraction isn't really that different from SAN or NFS 4.x object storage. I last experimented with iSCSI 20 years ago, mostly because it seemed so great for booting even less cooperative diskless hosts than Sun workstations over the network. But if I had a reliable TrueNAS and wanted to run oVirt, I'd just go with NFS. AFAIK oVirt was born on SAN but with SAN outside of oVirt's purvue. So if your iSCSI setup behaves like a SAN, oVirt should be easy to get going, but I've never tried myself. And the lack of tried and tested tutorials or videos from 20 different sources might be the reason oVirt didn't quite push out everybody else.

1 0

Oracle Virtualization Manager 4.5 anyone?
by Thomas Hoberg 29 Apr '24

29 Apr '24

Redhat's decision to shut down RHV caught Oracle pretty unprepared, I'd guess, who had just shut down their own vSphere clone in favor of a RHV clone a couple of years ago. Oracle is even less vocal about their "Oracle Virtualization" strategy, they don't even seem to have a proper naming convention or branding. But they have been pushing out OV releases without a publicly announced EOL almost a year behind Redhat for the last years. And after a 4.4 release in September 22, a few days ago on December 12th actually a release 4.5 was made public. I've operated oVirt 4.3 with significant quality issues for some years and failed to make oVirt 4.4 work with any degree of acceptable stability but Oracle's variant of 4.4 proved to be rather better than 4.3 on CentOS7 with no noticable bugs, especially in the Hyperconverged setup that I am using with GlusterFS. I assumed that this was because Oracle based their 4.4 in fact on RHV 4.4 and not oVirt, but since they're not telling, who knows? One issue with 4.4 was that Oracle is pushing their UE-Kernel and that created immediate issues e.g. with VDO missing modules for UEK and other stuff, but that was solved easily enough by using the RHEL kernel. With 4.5 Oracle obviously can't use RHV 4.5 as a base, because there is no such thing with RHV declared EOL and according to Oracle their 4.5 is based on oVirt 4.5.4, which made the quality of that release somewhat questionable, but perhaps they have spent the year that has passed since productively killing bugs... only to be caught by surprise again, I presume, by an oVirt release 4.5.5 on December 1st, that no one saw coming! Long story slightly shorter, I've been testing Oracle's 4.5 variant a bit and it's not without issues. But much worse, Oracle's variant of oVirt seems to be entirely without any community that I could find. Now oVirt has been a somewhat secret society for years, but compared to what's going on with Oracle this forum is teaming with life! So did I just not look around enough? Is there a secret lair where all those OV users are hiding? Anyhow, here is what I've tested so far and where I'd love to have some feedback: 1. Setting up a three node HCI cluster from scratch using OL8.9 and OV 4.5 Since I don't have extra physical hardware for a 3 node HCI I'm using VMware workstation 17.5 on a Workstation running Windows 2022, a test platform that has been working for all kinds of virtualization tests from VMware ESXi, via Xcp-ng and ovirt. Created three VMs with OL8.9 minimal and then installed OV 4.5. I used the UEK default kernels and then had an issue when Ansible is trying to create the (local) management engine: the VM simply could not reach the Oracle repo servers to install the packages inside the ME. Since that VM is entirely under the control of Ansible and no console access of any type is possible in that installation phase, I couldn't do diagnostics. But with 4.4 I used to have similar issues and there switching back to the Redhat kernel for the ME (and the hosts) resolved them. But with 4.5 it seems that UEK has become a baked-in dependency: the OV team doesn't even seem to do any testing with the Redhat kernel any more. Or not with the HCI setup, which has become deprecated somewhere in oVirt 4.4... Or not with the Cockpit wizard, which might be in a totally untested state, or.... Doing the same install on OL 8.9 with OV 4.4, however, did work just fine and I was even able to update to 4.5 afterwards, which was a nice surprise... ...that I could not repeat on my physical test farm using three Atoms. There switching to the UEK kernel on the hosts caused issues, hosts were becoming unresponsive, file systems inaccessible, even if they were perfectly fine at the Gluster CLI level and in the end the ME VM simply would not longer start. Switching back to the Redhat kernel resolved things there. In short, switching between the Redhat kernel and UEK, which should be 100% transparent to all things userland including hypervisors, doesn't work. But my attempts to go with a clean install of 4.5 on a Redhat kernel or UEK is also facing issues. So far the only thing that has worked was a single node HCI install using UEK and OV 4.5 and upgrading to OV 4.5 on a virtualized triple node OV 4.4 HCI cluster. Anyone else out there trying these things? I was mostly determined to move to Proxmox VE, but Oracle's OV 4.5 seemed to be handing a bit of a life-line to oVirt and the base architecture is just much more powerful (or less manual) than Proxmox, which doesn't have a management engine.

8 12

VMs don't receive unicast packets, can't reach gateway, don't get ARP responses.
by jeff.parrish＠dentsu.com 26 Apr '24

26 Apr '24

Hey All, I'm reaching out because I'm working on a PoC ovirt setup, and running into some networking issues from my VMs that I just have not been able to nail down or figure out. Setup: Three Hypervisors running on the ovirt image, one NIC on each is connected to a trunk port on our data network, the other is connected to our storage network. I'm just running a basic test setup with three logical networks - ovirtmgmt, stroage, vm_network the vNics have no network filtering enabled, I have disabled and stopped firewalld on the hypervisors as well. ovirtmgmt and storage have IPs on the hypervisor, vm_network does not have an IP (Thought I did try it with no difference.) I currently just have three VMs running since it's a PoC, one on each host - they are all connected to the vm_network, and all are unable to reach their gateway (a linux device), and unable to reach each other unless they are on the same hypervisor - this is the main issue I'm dealing with. Investigation done so far: Running a ping from the VM to the Gateway IP and capturing data along the flow gives me this: If I run a tcpdump on the hypervisor I can see ARP requests being broadcasted on the correct interface (vm_network, vnet0) from the VM looking for the gateway, however I never see any ARP response from the gateway on the hypervisor, and the VMs never update their arp table and never actually attempt to send any ICMP packets. If I run a tcpdump on the gateway device, I can see ARP requests from the VM coming in, and I can see the gateway giving a unicast ARP reply - again I never see that arp reply on the hypervisor, and I never see it reach the VM. Now where it gets a little bit interesting at least. if I run an arping from the gateway device to the VM - I get a response from the VM on the broadcast arp, but I do not get any responses (or see any traffic on the hypervisor) when it moves to a unicast arp probe ping. After doing this the VM will also update it's ARP table with the gateway. If I attempt to ping the gateway from the VM after this, it will send ICMP packets, which do reach the gateway, the gateway does respond... but just like all other unicast things I never see that response on the hypervisor, and it never reaches the VM. (This only lasts for a short period of time until the VM fails to get a new ARP response and removes the entry from the arp table.) There is no firewall setup on the gateway that would prevent any of this traffic, and everything with that gateway, vlan, and IPs work just fine outside of ovirt. The behavior between VMs (on different hypervisors) is exactly the same as VM to Gateway. Where arp requests go out, the VM on a different hypervisor will see and respond to the arp broadcast request, but I never see that unicast ARP reply make it to a tcpdump on the hypervisor. TLDR: Broadcast traffic reaches the VMs, unicast packets from the VM reach their destination (If they have a record for them in their arp table), unicast replies and unicast requests to the VMs never even show up in a tcpdump on the hypervisor. No firewalls in place to explain any of these issues. I am sure I likely screwed something up during the configuration of this, but I don't see my mistake anywhere

1 1

Re: [External] : Can't create shared disk after upgrade to 4.5.4
by mahenderjoshi31＠gmail.com 26 Apr '24

26 Apr '24

Hi, I have also faced the same issue. can anyone suggest the appropriate solution for this one.

2 1