- Users - oVirt List Archives

OVirt Self Hosted Enging VM issue
by linim＠gdls.com 31 Mar '23

31 Mar '23

Hello If this is a repeat discussion, I do apologize but I am looking to see if anyone else out there as had or heard of this issue happening? I am attempting to install a KVM hosts with a Self-Hosted-Engine. When it goes to install the OVIRT engine on the VM being created under its temporary IP, I am noticing that the VM will disconnect, and I lose all network connectivity on it. This is in attempt to install OLVM using the self-hosted-engine referencing oracle Doc ID 2933018.1. I have engaged oracle support, but I am just hoping that perhaps someone else has run into this issue and they can offer up avenues of investigation? If his is outside the scope of an oVirt discussion as it is OLVM, then this can be closed out.

1 0

Encrypted VNC request using SASL not maintained after VM migration
by Jon Sattelberger 29 Mar '23

29 Mar '23

I recently followed the instructions for enabling VNC encryption for FIPS enabled hosts [1]. The VNC console seem to be fine on the host where the VM is initially started (excluding noVNC in the browser). The qemu-kvm arguments are not maintained properly upon VM migration, declaring "password=on" in the -vnc argument. Subsequent VNC console requests will result in an authentication failure. SPICE seems to be fine. All hosts and the engine are FIPS enabled running oVirt-4.5.4-1.el8. Is there a way to maintain the absence of "password=on"after VM migation? Perhaps a hook in the interim. Initial VM start: -object {"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/vdsm/libvirt-vnc","endpoint":"server","verify-peer":false} -vnc 192.168.100.67:0,tls-creds=vnc-tls-creds0,sasl=on,audiodev=audio1 -k en-us Debug output from remote-viewer: (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.812: vncconnection.c Possible VeNCrypt sub-auth 263 (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.812: vncconnection.c Emit main context 12 (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.812: vncconnection.c Requested auth subtype 263 (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.813: vncconnection.c Waiting for VeNCrypt auth subtype (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.813: vncconnection.c Choose auth 263 (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.813: vncconnection.c Checking if credentials are needed (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.813: vncconnection.c No credentials required (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.813: vncconnection.c Read error Resource temporarily unavailable (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.841: vncconnection.c Do TLS handshake (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.944: vncconnection.c Checking if credentials are needed (remote-viewer:1495470): gtk-vnc-DEBUG: 12:51:55.944: vncconnection.c Want a TLS clientname ... snip ... Migrated VM: -object {"qom-type":"tls-creds-x509","id":"vnc-tls-creds0","dir":"/etc/pki/vdsm/libvirt-vnc","endpoint":"server","verify-peer":false} -vnc 192.168.100.68:0,password=on,tls-creds=vnc-tls-creds0,sasl=on,audiodev=audio1 -k en-us Debug output from remote-viewer: (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.487: vncconnection.c Possible VeNCrypt sub-auth 261 (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.487: vncconnection.c Emit main context 12 (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.488: vncconnection.c Requested auth subtype 261 (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.488: vncconnection.c Waiting for VeNCrypt auth subtype (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.488: vncconnection.c Choose auth 261 (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.488: vncconnection.c Checking if credentials are needed (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.488: vncconnection.c No credentials required ... snip ... (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.780: vncconnection.c Checking auth result (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.808: vncconnection.c Fail Authentication failed (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.808: vncconnection.c Error: Authentication failed (remote-viewer:1495270): gtk-vnc-DEBUG: 12:50:29.808: vncconnection.c Emit main context 16 (remote-viewer:1495270): virt-viewer-WARNING **: 12:50:29.808: vnc-session: got vnc error Authentication failed Thank you, Jon [1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/ht…

1 1

Disable "balancing" and authomatic migration
by Diego Ercolani 28 Mar '23

28 Mar '23

Hello, in my installation I have to use poor storage... the oVirt installation doesn't manage such a case and begin to "balance" and move VMs around... taking too many snapshots stressing a poor performance all the cluster mess up.... Why the vms don't go in "Pause" state but the cluster prefer to migrate things around messing up everything? This is a reference I found and for notice I'm disabling the auto-migration on every VM, hoping this help https://lists.ovirt.org/archives/list/users@ovirt.org/thread/24KQZFP2PCW462…

2 8

Failing "change Master storage domain" from gluster to iscsi
by Diego Ercolani 28 Mar '23

28 Mar '23

In the current release of ovirt (4.5.4) I'm currently experiencing a fail in change master storage domain from a gluster volume to everywhere. The GUI talk about a "general" error. watching the engine log: 2023-03-28 11:51:16,601Z WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-46) [] Unexpected return value: TaskStatus [code=331, message=value=Tar command failed: ({'reader': {'cmd': ['/usr/bin/tar', 'cf', '-', '--exclude=./lost+found', '-C', '/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/master', '.'], 'rc': 1, 'err': '/usr/bin/tar: ./tasks/20a9aa7f-80f5-403b-b296-ea95d9fd3f97: file changed as we read it\n/usr/bin/tar: ./tasks/87783efa-42ac-4cd9-bda5-ad68c59bb881/87783efa-42ac-4cd9-bda5-ad68c59bb881.task: file changed as we read it\n'}},) abortedcode=331] 2023-03-28 11:51:16,601Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-46) [] Failed in 'HSMGetAllTasksStatusesVDS' method Seeming that somewhat is changing file under the directory but: [vdsm@ovirt-node2 4745320f-bfc3-46c4-8849-b4fe8f1b2de6]$ /usr/bin/tar -cf - --exclude=./lost+found -C '/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/master' '.' > /tmp/tar.tar /usr/bin/tar: ./tasks/20a9aa7f-80f5-403b-b296-ea95d9fd3f97: file changed as we read it /usr/bin/tar: ./tasks: file changed as we read it [vdsm@ovirt-node2 master]$ find '/rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/master' -mtime -1 /rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/master/tasks [vdsm@ovirt-node2 master]$ ls -l /rhev/data-center/mnt/glusterSD/ovirt-node3.ovirt:_gv0/4745320f-bfc3-46c4-8849-b4fe8f1b2de6/master/ total 0 drwxr-xr-x. 6 vdsm kvm 182 Mar 28 11:51 tasks drwxr-xr-x. 2 vdsm kvm 6 Mar 26 20:36 vms [vdsm@ovirt-node2 master]$ date; stat tasks Tue Mar 28 12:04:06 UTC 2023 File: tasks Size: 182 Blocks: 0 IO Block: 131072 directory Device: 31h/49d Inode: 12434008067414313592 Links: 6 Access: (0755/drwxr-xr-x) Uid: ( 36/ vdsm) Gid: ( 36/ kvm) Context: system_u:object_r:fusefs_t:s0 Access: 2023-03-28 11:55:17.771046746 +0000 Modify: 2023-03-28 11:51:16.641145314 +0000 Change: 2023-03-28 11:51:16.641145314 +0000 Birth: - It seem the task directory isn't touched since

1 1

clock skew in hosted engine and VMs due to slow IO storage
by Diego Ercolani 28 Mar '23

28 Mar '23

I don't know why (but I suppose is related to storage speed) the virtual machines tend to present a skew in the clock from some days to a century forward (2177) I see in the journal of the engine: Mar 28 13:19:40 ovirt-engine.ovirt NetworkManager[1158]: <info> [1680009580.2045] dhcp4 (eth0): state changed new lease, address=192.168.123.20 Mar 28 13:24:40 ovirt-engine.ovirt NetworkManager[1158]: <info> [1680009880.2042] dhcp4 (eth0): state changed new lease, address=192.168.123.20 Mar 28 13:29:40 ovirt-engine.ovirt NetworkManager[1158]: <info> [1680010180.2039] dhcp4 (eth0): state changed new lease, address=192.168.123.20 Apr 01 08:15:42 ovirt-engine.ovirt chronyd[1072]: Forward time jump detected! Apr 01 08:15:42 ovirt-engine.ovirt NetworkManager[1158]: <info> [1680336942.4396] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds) Apr 01 08:15:42 ovirt-engine.ovirt chronyd[1072]: Can't synchronise: no selectable sources When this happens in the hosted-engine tipically: 1. the DWH became unconsistent as I stated here: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/KPW5FFKG3AI6EI… or https://lists.ovirt.org/archives/list/users@ovirt.org/thread/WUNZUSZ2ARRLGN… 2. the skew causes the engine to kick off the nodes that appears "down" in "connecting" state This compromises all the task in pending state and raise countermeasures to the ovirt-engine manager and also vdsm daemon. I currently tried to put in engine's crontab every 5 minutes a "hwclock --hctosys" as it seem the hwclock don't skew

1 0

How to enable the storage pool correctly
by ziyi Liu 28 Mar '23

28 Mar '23

The /var/ folder is full, I can't enter the web ui to set it, I can only use the command line mode vdsm-client StorageDomain activate vdsm-client StorageDomain attach vdsm-client StoragePool connect vdsm-client StoragePool connectStorageServer I have tried these commands and they all prompt message=Unknown pool id, pool not connected

1 0

Failing vm backup
by Giulio Casella 28 Mar '23

28 Mar '23

Hi, since yesterday backup for one of my VMs is failing. Backup is performed by Storware vprotect, based on CBT strategy. In ovirt events I can see: VDSM host03 command StartNbdServerVDS failed: Bitmap does not exist: "{'reason': 'Bitmap does not exist in /rhev/data-center/mnt/blockSD/459011cf-ebb6-46ff-831d-8ccfafd82c8a/images/a9ec8085-6ac3-4be0-bbd2-7752f7e29368/083f5dc5-9003-4908-96b1-5f750b5a4197', 'bitmap': '1b88a937-0ab8-4f6e-a117-bbd522e21448'}" Transfer was stopped by system. Reason: failed to create a signed image ticket. Backups for other VMs are working correctly. What could it be? Somehow corrupted image? I tried to migrate VM to other hypervisors, with no luck. Didn't tried to move disks (two, one for the OS, one for data) to another data domain, not so fast operation (about 2.8TB). The VM is working correctly, but Murphy's law states this machine is a fileserver with users' data and email :-( Any hint? TIA, gc -- Giulio Casella giulio at di.unimi.it System and network architect Computer Science Dept. - University of Milano

2 2

Fail to import VM from Export Domain
by carl langlois 27 Mar '23

27 Mar '23

Hi, Our main storage domain has failed and we are trying to recover it. On our system we also have an Export Domain that adds some clone vm in it. But when trying to import the vm from the export domain i always get a fail. 023-03-27 08:45:40,273-04 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (default task-31) [30eb771d-ccd6-4484-97bd-825aaf9b2443] Lock Acquired to object 'EngineLock:{exclusiveLocks='[flexlm_clone=VM_NAME, e36327c2-9019-4eb4-8309-4df0557961cb=VM]', sharedLocks='[99eae7e7-29b4-4226-907a-02011ca5eec1=REMOTE_VM]'}' 2023-03-27 08:47:28,697-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-35) [30eb771d-ccd6-4484-97bd-825aaf9b2443] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm flexlm_clone to Data Center Default, Cluster Default 2023-03-27 08:47:28,798-04 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engine-Thread-35) [30eb771d-ccd6-4484-97bd-825aaf9b2443] Lock freed to object 'EngineLock:{exclusiveLocks='[flexlm_clone=VM_NAME, e36327c2-9019-4eb4-8309-4df0557961cb=VM]', sharedLocks='[99eae7e7-29b4-4226-907a-02011ca5eec1=REMOTE_VM]'}' 2023-03-27 08:47:39,737-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-16) [30eb771d-ccd6-4484-97bd-825aaf9b2443] Trying to release exclusive lock which does not exist, lock key: 'flexlm_cloneVM_NAME' 2023-03-27 08:47:39,737-04 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-16) [30eb771d-ccd6-4484-97bd-825aaf9b2443] Lock freed to object 'EngineLock:{exclusiveLocks='[flexlm_clone=VM_NAME, e36327c2-9019-4eb4-8309-4df0557961cb=VM]', sharedLocks='[99eae7e7-29b4-4226-907a-02011ca5eec1=REMOTE_VM]'}' 2023-03-27 08:47:40,091-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-16) [30eb771d-ccd6-4484-97bd-825aaf9b2443] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm flexlm_clone to Data Center Default, Cluster Default I was able to import 1 of them. but all the others fail. Any hints to suggest? Regards Carl

1 0

Setup oVirt self hosted engine on Rocky LInux 8 using cockpit - stuck in deadlock
by dirk.schulz＠geniodata.de 27 Mar '23

27 Mar '23

Hi all, I have set up 3 servers in 3 data centers, each having one physical interface and a vlan interface parented by it. The connection between the 3 servers over the vlan interfaces (using private ip addresses) works (using icmp ping as the test). Now I want to turn them into an ovirt cluster creating the self hosted engine on the first server. I have - made sure the engine fqdn is in dns forward and reverse and in /etc/hosts - made sure that both interfaces have unique dns entries which can be resolved forward and reverse - made sure that both interfaces' fqdns are in /etc/hosts - made sure only the primary hostname (not fqdn) is in /etc/hostname, - made sure ipv6 is available on the physical interface, - made sure ipv6 method is "disabled" on the vlan interface, - set /usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/defaults/main.yml:he_force_ip4: true to make sure no ipv6 attempts to interfere. Now when I use cockpit's hosted engine wizard (not hyperconverged), i run into 2 opposing problems. If I set the FQDN in the "Advanced" sub pane to the FQDN of the vlan interface, the wizards gets stuck at "preparing VM" with "The resolved address doesn't resolve on the selected interface\n". If I set the FQDN in the "Advanced" sub pane to the FQDN of the physical interface, I get the same result. If i add the physical interfaces FQDN to the vlan ip address in /etc/hosts, i get "hostname 'x.y.z' doesn't uniquely match the interface 'enp5s0.4000' selected for the management bridge; it matches also interface with IP ['physical']. Please make sure that the hostname got from the interface for the management network resolves only there." So clearly separating the two interfaces namewise is mandatory. I tried to follow the ansible workflow step by step to see what it does. I seems the validate hostname is triggered twice, second time on filling in FQDN in "Advanced" sub pane - it succeeds with both hostnames (physiscal interface and vlan ip), but that does not prevent the "prepare VM" workflow in doing the same verification and failing, as far as I can see. This is where it happens: 2023-03-20 14:31:48,354+0100 DEBUG ansible on_any args TASK: ovirt.ovirt.hosted_engine_setup : Check the resolved address resolves on the selected interface kwargs is_conditional:False 2023-03-20 14:31:48,355+0100 DEBUG ansible on_any args localhost TASK: ovirt.ovirt.hosted_engine_setup : Check the resolved address resolves on the selected interface kwargs 2023-03-20 14:31:48,481+0100 DEBUG var changed: host "localhost" var "ansible_play_hosts" type "<class 'list'>" value: "[]" 2023-03-20 14:31:48,481+0100 DEBUG var changed: host "localhost" var "ansible_play_batch" type "<class 'list'>" value: "[]" 2023-03-20 14:31:48,481+0100 DEBUG var changed: host "localhost" var "play_hosts" type "<class 'list'>" value: "[]" 2023-03-20 14:31:48,481+0100 ERROR ansible failed { "ansible_host": "localhost", "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml", "ansible_result": { "_ansible_no_log": false, "changed": false, "msg": "The resolved address doesn't resolve on the selected interface\n" }, "ansible_task": "Check the resolved address resolves on the selected interface", "ansible_type": "task", "status": "FAILED", "task_duration": 0 } So I am really stuck there. I do not have any idea how and where to go on. I can try changing bits in the playbooks and parameters (like using "hostname -A" instead of "hostname -f" for the failing test), but that can't really be the idea - I am to new to this to run into a bug or similar, I will suspect I do overlook something. Any hint or help is appreciated. Cheers, Dirk

4 4

oVirt host deploy: disable multipath
by femi adegoke 27 Mar '23

27 Mar '23

https://bugzilla.redhat.com/show_bug.cgi?id=1472356 There are 2 drives in my install that oVirt is claiming as multipath. I have tried blacklisting at /etc/multipath/ but that hasn't worked. Any better solutions? I need these 2 drives to be non-multipathed.

4 19