Oracle Virtualization Manager 4.5 anyone?
by Thomas Hoberg
Redhat's decision to shut down RHV caught Oracle pretty unprepared, I'd guess, who had just shut down their own vSphere clone in favor of a RHV clone a couple of years ago.
Oracle is even less vocal about their "Oracle Virtualization" strategy, they don't even seem to have a proper naming convention or branding.
But they have been pushing out OV releases without a publicly announced EOL almost a year behind Redhat for the last years.
And after a 4.4 release in September 22, a few days ago on December 12th actually a release 4.5 was made public.
I've operated oVirt 4.3 with significant quality issues for some years and failed to make oVirt 4.4 work with any degree of acceptable stability but Oracle's variant of 4.4 proved to be rather better than 4.3 on CentOS7 with no noticable bugs, especially in the Hyperconverged setup that I am using with GlusterFS.
I assumed that this was because Oracle based their 4.4 in fact on RHV 4.4 and not oVirt, but since they're not telling, who knows?
One issue with 4.4 was that Oracle is pushing their UE-Kernel and that created immediate issues e.g. with VDO missing modules for UEK and other stuff, but that was solved easily enough by using the RHEL kernel.
With 4.5 Oracle obviously can't use RHV 4.5 as a base, because there is no such thing with RHV declared EOL and according to Oracle their 4.5 is based on oVirt 4.5.4, which made the quality of that release somewhat questionable, but perhaps they have spent the year that has passed since productively killing bugs... only to be caught by surprise again, I presume, by an oVirt release 4.5.5 on December 1st, that no one saw coming!
Long story slightly shorter, I've been testing Oracle's 4.5 variant a bit and it's not without issues.
But much worse, Oracle's variant of oVirt seems to be entirely without any community that I could find.
Now oVirt has been a somewhat secret society for years, but compared to what's going on with Oracle this forum is teaming with life!
So did I just not look around enough? Is there a secret lair where all those OV users are hiding?
Anyhow, here is what I've tested so far and where I'd love to have some feedback:
1. Setting up a three node HCI cluster from scratch using OL8.9 and OV 4.5
Since I don't have extra physical hardware for a 3 node HCI I'm using VMware workstation 17.5 on a Workstation running Windows 2022, a test platform that has been working for all kinds of virtualization tests from VMware ESXi, via Xcp-ng and ovirt.
Created three VMs with OL8.9 minimal and then installed OV 4.5. I used the UEK default kernels and then had an issue when Ansible is trying to create the (local) management engine: the VM simply could not reach the Oracle repo servers to install the packages inside the ME. Since that VM is entirely under the control of Ansible and no console access of any type is possible in that installation phase, I couldn't do diagnostics.
But with 4.4 I used to have similar issues and there switching back to the Redhat kernel for the ME (and the hosts) resolved them.
But with 4.5 it seems that UEK has become a baked-in dependency: the OV team doesn't even seem to do any testing with the Redhat kernel any more. Or not with the HCI setup, which has become deprecated somewhere in oVirt 4.4... Or not with the Cockpit wizard, which might be in a totally untested state, or....
Doing the same install on OL 8.9 with OV 4.4, however, did work just fine and I was even able to update to 4.5 afterwards, which was a nice surprise...
...that I could not repeat on my physical test farm using three Atoms. There switching to the UEK kernel on the hosts caused issues, hosts were becoming unresponsive, file systems inaccessible, even if they were perfectly fine at the Gluster CLI level and in the end the ME VM simply would not longer start. Switching back to the Redhat kernel resolved things there.
In short, switching between the Redhat kernel and UEK, which should be 100% transparent to all things userland including hypervisors, doesn't work.
But my attempts to go with a clean install of 4.5 on a Redhat kernel or UEK is also facing issues. So far the only thing that has worked was a single node HCI install using UEK and OV 4.5 and upgrading to OV 4.5 on a virtualized triple node OV 4.4 HCI cluster.
Anyone else out there trying these things?
I was mostly determined to move to Proxmox VE, but Oracle's OV 4.5 seemed to be handing a bit of a life-line to oVirt and the base architecture is just much more powerful (or less manual) than Proxmox, which doesn't have a management engine.
10 months, 2 weeks
VMs don't receive unicast packets, can't reach gateway, don't get ARP responses.
by jeff.parrish＠dentsu.com
Hey All,
I'm reaching out because I'm working on a PoC ovirt setup, and running into some networking issues from my VMs that I just have not been able to nail down or figure out.
Setup:
Three Hypervisors running on the ovirt image, one NIC on each is connected to a trunk port on our data network, the other is connected to our storage network.
I'm just running a basic test setup with three logical networks - ovirtmgmt, stroage, vm_network the vNics have no network filtering enabled, I have disabled and stopped firewalld on the hypervisors as well. ovirtmgmt and storage have IPs on the hypervisor, vm_network does not have an IP (Thought I did try it with no difference.)
I currently just have three VMs running since it's a PoC, one on each host - they are all connected to the vm_network, and all are unable to reach their gateway (a linux device), and unable to reach each other unless they are on the same hypervisor - this is the main issue I'm dealing with.
Investigation done so far:
Running a ping from the VM to the Gateway IP and capturing data along the flow gives me this:
If I run a tcpdump on the hypervisor I can see ARP requests being broadcasted on the correct interface (vm_network, vnet0) from the VM looking for the gateway, however I never see any ARP response from the gateway on the hypervisor, and the VMs never update their arp table and never actually attempt to send any ICMP packets.
If I run a tcpdump on the gateway device, I can see ARP requests from the VM coming in, and I can see the gateway giving a unicast ARP reply - again I never see that arp reply on the hypervisor, and I never see it reach the VM.
Now where it gets a little bit interesting at least.
if I run an arping from the gateway device to the VM - I get a response from the VM on the broadcast arp, but I do not get any responses (or see any traffic on the hypervisor) when it moves to a unicast arp probe ping. After doing this the VM will also update it's ARP table with the gateway. If I attempt to ping the gateway from the VM after this, it will send ICMP packets, which do reach the gateway, the gateway does respond... but just like all other unicast things I never see that response on the hypervisor, and it never reaches the VM. (This only lasts for a short period of time until the VM fails to get a new ARP response and removes the entry from the arp table.)
There is no firewall setup on the gateway that would prevent any of this traffic, and everything with that gateway, vlan, and IPs work just fine outside of ovirt.
The behavior between VMs (on different hypervisors) is exactly the same as VM to Gateway. Where arp requests go out, the VM on a different hypervisor will see and respond to the arp broadcast request, but I never see that unicast ARP reply make it to a tcpdump on the hypervisor.
TLDR:
Broadcast traffic reaches the VMs, unicast packets from the VM reach their destination (If they have a record for them in their arp table), unicast replies and unicast requests to the VMs never even show up in a tcpdump on the hypervisor. No firewalls in place to explain any of these issues.
I am sure I likely screwed something up during the configuration of this, but I don't see my mistake anywhere
10 months, 3 weeks
hosted-engine --deploy - trigger storage rescan
by Angus Clarke
Hello
OLVM 4.5 - SHE install
Our storage guy messed up the FC storage allocation -> when the SHE install got to ask about storage for the hosted_engine the expected LUN was not in the list.
It took the process around an hour to get to this point so I didn't want to restart from the beginning, found that issuing a SIGHUP to the python3 process from another screen made it drop back to asking about storage again and went on to list the correctly provisioned LUNs - very useful but I didn't find mention of it in the man pages; sharing for prosperity.
Regards
Angus
...
[ INFO ] ok: [localhost]
The following luns have been found on the requested target:
[1] 3600c0ff00052f2cb80912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[2] 3600c0ff00052f2cb8a912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[3] 3600c0ff00052f2cb8e912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[4] 3600c0ff00052f2cb92912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[5] 3600c0ff00052f2cb96912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[6] 3600c0ff00052f2d781912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[7] 3600c0ff00052f2d78b912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[8] 3600c0ff00052f2d78f912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[9] 3600c0ff00052f2d793912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
Please select the destination LUN (1, 2, 3, 4, 5, 6, 7, 8, 9) [1]:0
[ ERROR ] Invalid value
Please select the destination LUN (1, 2, 3, 4, 5, 6, 7, 8, 9) [1]:?
[ ERROR ] Invalid value
Please select the destination LUN (1, 2, 3, 4, 5, 6, 7, 8, 9) [1]:r
[ ERROR ] Invalid value
Please select the destination LUN (1, 2, 3, 4, 5, 6, 7, 8, 9) [1]:[ ERROR ] Unable to get target list
Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]:fc
[ INFO ] Getting Fibre Channel LUNs list
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Force facts gathering]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get Fibre Channel LUNs]
[ INFO ] ok: [localhost]
The following luns have been found on the requested target:
[1] 3600c0ff00052f2cb74912b6601000000 465.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[2] 3600c0ff00052f2cb80912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[3] 3600c0ff00052f2cb8a912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[4] 3600c0ff00052f2cb8e912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[5] 3600c0ff00052f2cb92912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[6] 3600c0ff00052f2cb96912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[7] 3600c0ff00052f2d771912b6601000000 465.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[8] 3600c0ff00052f2d775912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[9] 3600c0ff00052f2d781912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[10] 3600c0ff00052f2d78b912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[11] 3600c0ff00052f2d78f912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
[12] 3600c0ff00052f2d793912b6601000000 1862.0GiB HPE MSA 2050 SAN
status: free, paths: 4 active
Please select the destination LUN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12) [1]:7
[ INFO ] FC discard after delete is enabled
[ INFO ] Creating Storage Domain
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO ] ok: [localhost]
...
[ INFO ] Stage: Termination
[ INFO ] Hosted Engine successfully deployed
10 months, 3 weeks
Ovirt Hyperconverged
by eevans＠digitaldatatechs.com
CentOS 7 3 servers
Gluster deployed without a problem.
Hosted engine deployment failsL
[ ERROR ] fatal: [localhost]: FAILED! => {"reason": "conflicting action statements: fail, msg\n\nThe error appears to be in '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/pre_checks/validate_mac_address.yml': line 14, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n register: he_vm_mac_addr\n - name: Fail if MAC address structure is incorrect\n ^ here\n"}
I'm not dsure why this error occurs. I tried the same file from a different server and even tried a different server.
Something is wrong with this file structure.
Anyhelp is appreviated.
Eric Evans
10 months, 3 weeks
Re: Migration fails after upgrading target host to oVirt Node 4.5.5
by Jonas
Thanks for your response! I guess my path will be to migrate to el9 ovirt-node then.
Am 25. April 2024 08:18:57 MESZ schrieb "Nathanaël Blanchet" <blanchet(a)abes.fr>:
>Hello,
>I already mentioned this issue, it is because of el8 4.5.5 embedded qemu version that is not compatible with 4.5.4 one. If all your hosts are in 4.5.5 migration is ok but it involves that you shutdown your production vm.
>By the way, the solution if your hardware is compatible (HBA not supported anymore) is to use el9 based ovirt-node 4.5.5 that support migration from any qemu versions.
>
>Le 24 avr. 2024 20:59, jonas(a)rabe.ch a écrit :
>Hello all
>
>After upgrading one node from 4.5.4 to 4.5.5 the migration fails on some of the VMs. I believe the error is with qemu-kvm according to the logs below.
>
>Downgrading the packages seems not to be possible with oVirt Node, the only solution so far was rebooting back to 4.5.4 (thanks to imgbase). I also think it is related to https://lists.ovirt.org/archives/list/users@ovirt.org/thread/GGMEGFI4OGZ5..., but that thread is already several months old. Has someone had the same experience and found a solution?
>
>Packages:
>Before (4.5.4):
>- `current_layer: ovirt-node-ng-4.5.4-0.20221206.0+1`
>- Kernel: 4.18.0-408.el8.x86_64
>- qemu-kvm.x86_64: 15:6.2.0-20.module_el8.7.0+1218+f626c2ff.1
>- vdsm.x86_64: 4.50.3.4-1.el8
>- libvirt.x86_64: 8.0.0-10.module_el8.7.0+1218+f626c2ff
>- ovirt-host.x86_64: 4.5.0-3.el8
>
>After (4.5.5):
>- `current_layer: ovirt-node-ng-4.5.5-0.20231130.0+1`
>- Kernel: 4.18.0-526.el8.x86_64
>- qemu-kvm.x86_64: 15:6.2.0-41.module_el8+690+3a5f4f4f
>- vdsm.x86_64: 4.50.5.1-1.el8
>- libvirt.x86_64: 8.0.0-22.module_el8+596+27e96798
>- ovirt-host.x86_64: 4.5.0-3.el8
>
>/var/log/libvirt/qemu/vm-0008.log:
>2024-04-24 18:05:59.214+0000: starting up libvirt version: 8.0.0, package: 22.module_el8+596+27e96798 (builder(a)centos.org, 2023-07-31-14:36:36, ), qemu version: 6.2.0qemu-kvm-6.2.0-41.module_el8+690+3a5f4f4f, kernel: 4.18.0-526.el8.x86_64, hostname: server-007.XXX.YYY
>LC_ALL=C \
>PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
>HOME=/var/lib/libvirt/qemu/domain-14-vm-0008 \
>XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-14-vm-0008/.local/share \
>XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-14-vm-0008/.cache \
>XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-14-vm-0008/.config \
>/usr/libexec/qemu-kvm \
>-name guest=vm-0008,debug-threads=on \
>-S \
>-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-14-vm-0008/master-key.aes"}' \
>-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off \
>-accel kvm \
>-cpu Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on \
>-m size=8388608k,slots=16,maxmem=16777216k \
>-overcommit mem-lock=off \
>-smp 2,maxcpus=16,sockets=16,dies=1,cores=1,threads=1 \
>-numa node,nodeid=0,cpus=0-15,mem=8192 \
>-uuid 790499a6-1391-4c03-b270-e47ebfb851ff \
>-smbios type=1,manufacturer=oVirt,product=RHEL,version=8.7.2206.0-1.el8,serial=00000000-0000-0000-0000-ac1f6bcbc1de,uuid=790499a6-1391-4c03-b270-e47ebfb851ff,family=oVirt \
>-no-user-config \
>-nodefaults \
>-chardev socket,id=charmonitor,fd=50,server=on,wait=off \
>-mon chardev=charmonitor,id=monitor,mode=control \
>-rtc base=2024-04-24T18:05:58,driftfix=slew \
>-global kvm-pit.lost_tick_policy=delay \
>-no-hpet \
>-no-shutdown \
>-global PIIX4_PM.disable_s3=1 \
>-global PIIX4_PM.disable_s4=1 \
>-boot strict=on \
>-device piix3-usb-uhci,id=ua-ce46234b-4849-495e-81be-f29ac9f354a9,bus=pci.0,addr=0x1.0x2 \
>-device virtio-scsi-pci,id=ua-b36161d3-1a42-496e-8b3b-7fb4fc5844b8,bus=pci.0,addr=0x3 \
>-device virtio-serial-pci,id=ua-6504edef-b6c0-4812-a556-63517376c49e,max_ports=16,bus=pci.0,addr=0x4 \
>-device ide-cd,bus=ide.1,unit=0,id=ua-2bb69b71-f9a3-4c95-b3af-dd7ff63d249f,werror=report,rerror=report \
>-blockdev '{"driver":"file","filename":"/run/vdsm/payload/790499a6-1391-4c03-b270-e47ebfb851ff.img","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
>-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \
>-device ide-cd,bus=ide.1,unit=1,drive=libvirt-3-format,id=ua-b1b8cc32-d221-4c83-8b6d-41c953c731bd,werror=report,rerror=report \
>-blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/glusterSD/server-005.XXX.YYY:_tier1-ovirt-data-01/a047cdc3-1138-406f-89c8-efdc3924ce67/images/a79e4b9e-25ef-444a-87ab-eab1ba2f6eee/2bda74c4-019e-4dc2-ad8d-d867c869e784","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
>-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
>-device scsi-hd,bus=ua-b36161d3-1a42-496e-8b3b-7fb4fc5844b8.0,channel=0,scsi-id=0,lun=0,device_id=a79e4b9e-25ef-444a-87ab-eab1ba2f6eee,drive=libvirt-2-format,id=ua-a79e4b9e-25ef-444a-87ab-eab1ba2f6eee,bootindex=1,write-cache=on,serial=a79e4b9e-25ef-444a-87ab-eab1ba2f6eee,werror=stop,rerror=stop \
>-blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/glusterSD/server-005.XXX.YYY:_tier1-ovirt-data-01/a047cdc3-1138-406f-89c8-efdc3924ce67/images/c100660a-19c2-474f-af92-6a5bfb5a6698/f15cea6a-d42a-4db6-9f82-4f2c7ca4295d","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
>-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
>-device scsi-hd,bus=ua-b36161d3-1a42-496e-8b3b-7fb4fc5844b8.0,channel=0,scsi-id=0,lun=1,device_id=c100660a-19c2-474f-af92-6a5bfb5a6698,drive=libvirt-1-format,id=ua-c100660a-19c2-474f-af92-6a5bfb5a6698,write-cache=on,serial=c100660a-19c2-474f-af92-6a5bfb5a6698,werror=stop,rerror=stop \
>-netdev tap,fds=51:53,id=hostua-9f570dc0-c9a6-4f46-9283-25468ace64d1,vhost=on,vhostfds=54:55 \
>-device virtio-net-pci,mq=on,vectors=6,host_mtu=1500,netdev=hostua-9f570dc0-c9a6-4f46-9283-25468ace64d1,id=ua-9f570dc0-c9a6-4f46-9283-25468ace64d1,mac=00:1a:4a:16:01:58,bus=pci.0,addr=0x6 \
>-netdev tap,fds=56:57,id=hostua-34d14d9d-0427-47e0-a2f4-57c9450c8ecc,vhost=on,vhostfds=58:59 \
>-device virtio-net-pci,mq=on,vectors=6,host_mtu=1500,netdev=hostua-34d14d9d-0427-47e0-a2f4-57c9450c8ecc,id=ua-34d14d9d-0427-47e0-a2f4-57c9450c8ecc,mac=00:1a:4a:16:01:59,bus=pci.0,addr=0x7 \
>-chardev socket,id=charchannel0,fd=40,server=on,wait=off \
>-device virtserialport,bus=ua-6504edef-b6c0-4812-a556-63517376c49e.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 \
>-chardev socket,id=charchannel1,fd=45,server=on,wait=off \
>-device virtserialport,bus=ua-6504edef-b6c0-4812-a556-63517376c49e.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \
>-chardev spicevmc,id=charchannel2,name=vdagent \
>-device virtserialport,bus=ua-6504edef-b6c0-4812-a556-63517376c49e.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 \
>-audiodev '{"id":"audio1","driver":"spice"}' \
>-spice port=5900,tls-port=5901,addr=10.128.16.7,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on \
>-device qxl-vga,id=ua-c2f62823-2d82-4aac-b300-72ceafd6924d,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
>-incoming defer \
>-device virtio-balloon-pci,id=ua-0a31723e-6654-4ab8-995d-e0d314964c24,bus=pci.0,addr=0x5 \
>-device vmcoreinfo \
>-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
>-msg timestamp=on
>2024-04-24 18:05:59.214+0000: Domain id=14 is tainted: hook-script
>2024-04-24 18:05:59.215+0000: Domain id=14 is tainted: custom-hypervisor-feature
>2024-04-24T18:05:59.313461Z qemu-kvm: -numa node,nodeid=0,cpus=0-15,mem=8192: warning: Parameter -numa node,mem is deprecated, use -numa node,memdev instead
>2024-04-24T18:06:04.827095Z qemu-kvm: Missing section footer for 0000:00:01.3/piix4_pm
>2024-04-24T18:06:04.827282Z qemu-kvm: load of migration failed: Invalid argument
>2024-04-24 18:06:05.008+0000: shutting down, reason=failed
>_______________________________________________
>Users mailing list -- users(a)ovirt.org
>To unsubscribe send an email to users-leave(a)ovirt.org
>Privacy Statement: https://www.ovirt.org/privacy-policy.html
>oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
>List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R7FADODOTKJ...
10 months, 3 weeks
Migration fails after upgrading target host to oVirt Node 4.5.5
by jonas＠rabe.ch
Hello all
After upgrading one node from 4.5.4 to 4.5.5 the migration fails on some of the VMs. I believe the error is with qemu-kvm according to the logs below.
Downgrading the packages seems not to be possible with oVirt Node, the only solution so far was rebooting back to 4.5.4 (thanks to imgbase). I also think it is related to https://lists.ovirt.org/archives/list/users@ovirt.org/thread/GGMEGFI4OGZ5..., but that thread is already several months old. Has someone had the same experience and found a solution?
Packages:
Before (4.5.4):
- `current_layer: ovirt-node-ng-4.5.4-0.20221206.0+1`
- Kernel: 4.18.0-408.el8.x86_64
- qemu-kvm.x86_64: 15:6.2.0-20.module_el8.7.0+1218+f626c2ff.1
- vdsm.x86_64: 4.50.3.4-1.el8
- libvirt.x86_64: 8.0.0-10.module_el8.7.0+1218+f626c2ff
- ovirt-host.x86_64: 4.5.0-3.el8
After (4.5.5):
- `current_layer: ovirt-node-ng-4.5.5-0.20231130.0+1`
- Kernel: 4.18.0-526.el8.x86_64
- qemu-kvm.x86_64: 15:6.2.0-41.module_el8+690+3a5f4f4f
- vdsm.x86_64: 4.50.5.1-1.el8
- libvirt.x86_64: 8.0.0-22.module_el8+596+27e96798
- ovirt-host.x86_64: 4.5.0-3.el8
/var/log/libvirt/qemu/vm-0008.log:
2024-04-24 18:05:59.214+0000: starting up libvirt version: 8.0.0, package: 22.module_el8+596+27e96798 (builder(a)centos.org, 2023-07-31-14:36:36, ), qemu version: 6.2.0qemu-kvm-6.2.0-41.module_el8+690+3a5f4f4f, kernel: 4.18.0-526.el8.x86_64, hostname: server-007.XXX.YYY
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-14-vm-0008 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-14-vm-0008/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-14-vm-0008/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-14-vm-0008/.config \
/usr/libexec/qemu-kvm \
-name guest=vm-0008,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-14-vm-0008/master-key.aes"}' \
-machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off \
-accel kvm \
-cpu Cascadelake-Server-noTSX,mpx=off,hypervisor=on,pku=on,arch-capabilities=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on \
-m size=8388608k,slots=16,maxmem=16777216k \
-overcommit mem-lock=off \
-smp 2,maxcpus=16,sockets=16,dies=1,cores=1,threads=1 \
-numa node,nodeid=0,cpus=0-15,mem=8192 \
-uuid 790499a6-1391-4c03-b270-e47ebfb851ff \
-smbios type=1,manufacturer=oVirt,product=RHEL,version=8.7.2206.0-1.el8,serial=00000000-0000-0000-0000-ac1f6bcbc1de,uuid=790499a6-1391-4c03-b270-e47ebfb851ff,family=oVirt \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=50,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=2024-04-24T18:05:58,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot strict=on \
-device piix3-usb-uhci,id=ua-ce46234b-4849-495e-81be-f29ac9f354a9,bus=pci.0,addr=0x1.0x2 \
-device virtio-scsi-pci,id=ua-b36161d3-1a42-496e-8b3b-7fb4fc5844b8,bus=pci.0,addr=0x3 \
-device virtio-serial-pci,id=ua-6504edef-b6c0-4812-a556-63517376c49e,max_ports=16,bus=pci.0,addr=0x4 \
-device ide-cd,bus=ide.1,unit=0,id=ua-2bb69b71-f9a3-4c95-b3af-dd7ff63d249f,werror=report,rerror=report \
-blockdev '{"driver":"file","filename":"/run/vdsm/payload/790499a6-1391-4c03-b270-e47ebfb851ff.img","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \
-device ide-cd,bus=ide.1,unit=1,drive=libvirt-3-format,id=ua-b1b8cc32-d221-4c83-8b6d-41c953c731bd,werror=report,rerror=report \
-blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/glusterSD/server-005.XXX.YYY:_tier1-ovirt-data-01/a047cdc3-1138-406f-89c8-efdc3924ce67/images/a79e4b9e-25ef-444a-87ab-eab1ba2f6eee/2bda74c4-019e-4dc2-ad8d-d867c869e784","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device scsi-hd,bus=ua-b36161d3-1a42-496e-8b3b-7fb4fc5844b8.0,channel=0,scsi-id=0,lun=0,device_id=a79e4b9e-25ef-444a-87ab-eab1ba2f6eee,drive=libvirt-2-format,id=ua-a79e4b9e-25ef-444a-87ab-eab1ba2f6eee,bootindex=1,write-cache=on,serial=a79e4b9e-25ef-444a-87ab-eab1ba2f6eee,werror=stop,rerror=stop \
-blockdev '{"driver":"file","filename":"/rhev/data-center/mnt/glusterSD/server-005.XXX.YYY:_tier1-ovirt-data-01/a047cdc3-1138-406f-89c8-efdc3924ce67/images/c100660a-19c2-474f-af92-6a5bfb5a6698/f15cea6a-d42a-4db6-9f82-4f2c7ca4295d","aio":"threads","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device scsi-hd,bus=ua-b36161d3-1a42-496e-8b3b-7fb4fc5844b8.0,channel=0,scsi-id=0,lun=1,device_id=c100660a-19c2-474f-af92-6a5bfb5a6698,drive=libvirt-1-format,id=ua-c100660a-19c2-474f-af92-6a5bfb5a6698,write-cache=on,serial=c100660a-19c2-474f-af92-6a5bfb5a6698,werror=stop,rerror=stop \
-netdev tap,fds=51:53,id=hostua-9f570dc0-c9a6-4f46-9283-25468ace64d1,vhost=on,vhostfds=54:55 \
-device virtio-net-pci,mq=on,vectors=6,host_mtu=1500,netdev=hostua-9f570dc0-c9a6-4f46-9283-25468ace64d1,id=ua-9f570dc0-c9a6-4f46-9283-25468ace64d1,mac=00:1a:4a:16:01:58,bus=pci.0,addr=0x6 \
-netdev tap,fds=56:57,id=hostua-34d14d9d-0427-47e0-a2f4-57c9450c8ecc,vhost=on,vhostfds=58:59 \
-device virtio-net-pci,mq=on,vectors=6,host_mtu=1500,netdev=hostua-34d14d9d-0427-47e0-a2f4-57c9450c8ecc,id=ua-34d14d9d-0427-47e0-a2f4-57c9450c8ecc,mac=00:1a:4a:16:01:59,bus=pci.0,addr=0x7 \
-chardev socket,id=charchannel0,fd=40,server=on,wait=off \
-device virtserialport,bus=ua-6504edef-b6c0-4812-a556-63517376c49e.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 \
-chardev socket,id=charchannel1,fd=45,server=on,wait=off \
-device virtserialport,bus=ua-6504edef-b6c0-4812-a556-63517376c49e.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 \
-chardev spicevmc,id=charchannel2,name=vdagent \
-device virtserialport,bus=ua-6504edef-b6c0-4812-a556-63517376c49e.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 \
-audiodev '{"id":"audio1","driver":"spice"}' \
-spice port=5900,tls-port=5901,addr=10.128.16.7,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on \
-device qxl-vga,id=ua-c2f62823-2d82-4aac-b300-72ceafd6924d,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-incoming defer \
-device virtio-balloon-pci,id=ua-0a31723e-6654-4ab8-995d-e0d314964c24,bus=pci.0,addr=0x5 \
-device vmcoreinfo \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2024-04-24 18:05:59.214+0000: Domain id=14 is tainted: hook-script
2024-04-24 18:05:59.215+0000: Domain id=14 is tainted: custom-hypervisor-feature
2024-04-24T18:05:59.313461Z qemu-kvm: -numa node,nodeid=0,cpus=0-15,mem=8192: warning: Parameter -numa node,mem is deprecated, use -numa node,memdev instead
2024-04-24T18:06:04.827095Z qemu-kvm: Missing section footer for 0000:00:01.3/piix4_pm
2024-04-24T18:06:04.827282Z qemu-kvm: load of migration failed: Invalid argument
2024-04-24 18:06:05.008+0000: shutting down, reason=failed
10 months, 3 weeks
oVirt 4.5.6
by tasnadi.peter＠kifu.gov.hu
Hi,
When updating HE I noticed that version 4.5.6 is available, but on this link the 4.5.6 ISO NG image is not available.
https://resources.ovirt.org/pub/ovirt-4.5/iso/ovirt-node-ng-installer/
Even when updating the oVirt node, version 4.5.6 is not available.
The oVirt node was installed using the ng installer.
These versions have always been released once.
What is the reason for this?
Can I upgrade HE to 4.5.6 regardless?
Could the version difference be a problem?
Thanks for the help!
10 months, 3 weeks
Host deploy fails in 4.5.6
by Lars Stolpe
Hello,
I do have the same issue as some others and tryed to follow the instruction to downgrade ansible-core.
Active versions:
oVirt: 4.5.6-1
ansible-core: 2.16.3-2
It is not possible to downgrade to 2.12 due to dependency:
Error:
Problem: problem with installed package ovirt-engine-4.5.6-1.el8.noarch
- package ovirt-engine-4.5.6-1.el8.noarch from @System requires ansible-core >= 2.13.0, but none of the providers can be installed
the error message in deploy log:
"stdout" : "fatal: [blxbwf855]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"192.168.1.229\", \"blxbwf855\"], \"delta\": \"0:00:02.749603\", \"end\": \"2024-04-23 11:26:13.304995\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-04-23 11:26:10.555392\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\nKeyError: 'blxbwf855'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*args)\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\\n
raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: blxbwf855\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\", \" return networks[net_name]\", \"KeyError: 'blxbwf855'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_caps(), net_name))\", \" File \\\"/usr/lib/python3.6/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: blxbwf855\"], \"stdout\": \"\", \"stdout_lines\": []}",
I hope someone can help.
10 months, 3 weeks
Changing disk QoS causes segfault with IO-Threads enabled (oVirt 4.3.0.4-1.el7)
by jloh＠squiz.net
We recently upgraded to 4.3.0 and have found that when changing disk QoS settings on VMs whilst IO-Threads is enabled causes them to segfault and the VM to reboot. We've been able to replicate this across several VMs. VMs with IO-Threads disabled/turned off do not segfault when changing the QoS.
Mar 1 11:49:06 srvXX kernel: IO iothread1[30468]: segfault at fffffffffffffff8 ip 0000557649f2bd24 sp 00007f80de832f60 error 5 in qemu-kvm[5576498dd000+a03000]
Mar 1 11:49:06 srvXX abrt-hook-ccpp: invalid number 'iothread1'
Mar 1 11:49:11 srvXX libvirtd: 2019-03-01 00:49:11.116+0000: 13365: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer
Happy to supply some more logs to someone if they'll help but just wondering whether anyone else has experienced this or knows of a current fix other than turning io-threads off.
Cheers.
10 months, 3 weeks
