- Users - oVirt List Archives

Update Issue
by Gary Pedretty 21 Nov '21

21 Nov '21

After doing the latest updates on one of my CentOS 8 Stream hosts, it will non longer allow VMs to be migrated to it. VMs can be started on the host, but then have no network access. The hosted engine can be started on the host, but again no network access. I have tried removing the host, rebuilding the OS from scratch and re-adding it, with no change. It shows up and you do not get any error messages that I can tell when a migrate fails, it just shows that it failed. The updated host has the following versions, only KVM and LIBVIRT appear to be different. I tried downgrading KVM and LIBVIRT, but the migration still fails. I have avoided trying up update the other hosts since I cannot migrate any of my VMs to the one host that I already tried to update. RHEL - 8.6 - 1.el8 OS Description: CentOS Stream 8 Kernel Version: 4.18.0 - 348.el8.x86_64 KVM Version: 6.1.0 - 4.module_el8.6.0+983+a7505f3f LIBVIRT Version: libvirt-7.9.0-1.module_el8.6.0+983+a7505f3f VDSM Version: vdsm-4.40.90.4-1.el8 The other two hosts which have not been updated and still work normally have RHEL - 8.6 - 1.el8 OS Description: CentOS Stream 8 Kernel Version: 4.18.0 - 348.el8.x86_64 KVM Version: 6.0.0 - 33.el8s LIBVIRT Version: libvirt-7.6.0-4.el8s VDSM Version: vdsm-4.40.90.4-1.el8 The engine log from an attempted migrations is as follows. 2021-11-17 21:12:46,099-09 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-21) [c0c87da3-a8aa-4f8e-ab9c-1e2fc05babac] Running command: MigrateVmToServerCommand internal: false. Entities affected : ID: fa1a2d6b-99cb-42bd-a343-91f314d5f47b Type: VMAction group MIGRATE_VM with role type USER 2021-11-17 21:12:46,134-09 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-21) [c0c87da3-a8aa-4f8e-ab9c-1e2fc05babac] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='6ee16602-c686-471a-9f65-e5952b813672', vmId='fa1a2d6b-99cb-42bd-a343-91f314d5f47b', srcHost='ravn-kvm-8.ravnalaska.net', dstVdsId='10491335-e2e1-49f2-96c3-79331535542b', dstHost='ravn-kvm-9.ravnalaska.net:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', migrateEncrypted='false', consoleAddress='null', maxBandwidth='1250', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='10.9.24.79'}), log id: 3522dfb9 2021-11-17 21:12:46,134-09 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-21) [c0c87da3-a8aa-4f8e-ab9c-1e2fc05babac] START, MigrateBrokerVDSCommand(HostName = ravn-kvm-8.ravnalaska.net, MigrateVDSCommandParameters:{hostId='6ee16602-c686-471a-9f65-e5952b813672', vmId='fa1a2d6b-99cb-42bd-a343-91f314d5f47b', srcHost='ravn-kvm-8.ravnalaska.net', dstVdsId='10491335-e2e1-49f2-96c3-79331535542b', dstHost='ravn-kvm-9.ravnalaska.net:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', migrateEncrypted='false', consoleAddress='null', maxBandwidth='1250', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='10.9.24.79'}), log id: 23f98865 2021-11-17 21:12:46,141-09 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-21) [c0c87da3-a8aa-4f8e-ab9c-1e2fc05babac] FINISH, MigrateBrokerVDSCommand, return: , log id: 23f98865 2021-11-17 21:12:46,143-09 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-21) [c0c87da3-a8aa-4f8e-ab9c-1e2fc05babac] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 3522dfb9 2021-11-17 21:12:46,149-09 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-21) [c0c87da3-a8aa-4f8e-ab9c-1e2fc05babac] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: ns8, Source: ravn-kvm-8.ravnalaska.net, Destination: ravn-kvm-9.ravnalaska.net, User: admin@internal-authz). 2021-11-17 21:12:49,052-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-36) [] Fetched 2 VMs from VDS '10491335-e2e1-49f2-96c3-79331535542b' 2021-11-17 21:12:49,053-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-36) [] VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b' is migrating to VDS '10491335-e2e1-49f2-96c3-79331535542b'(ravn-kvm-9.ravnalaska.net) ignoring it in the refresh until migration is done 2021-11-17 21:12:54,314-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b' was reported as Down on VDS '10491335-e2e1-49f2-96c3-79331535542b'(ravn-kvm-9.ravnalaska.net) 2021-11-17 21:12:54,314-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b'(ns8) was unexpectedly detected as 'Down' on VDS '10491335-e2e1-49f2-96c3-79331535542b'(ravn-kvm-9.ravnalaska.net) (expected on '6ee16602-c686-471a-9f65-e5952b813672') 2021-11-17 21:12:54,315-09 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-21) [] START, DestroyVDSCommand(HostName = ravn-kvm-9.ravnalaska.net, DestroyVmVDSCommandParameters:{hostId='10491335-e2e1-49f2-96c3-79331535542b', vmId='fa1a2d6b-99cb-42bd-a343-91f314d5f47b', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 389ad865 2021-11-17 21:12:54,596-09 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-21) [] FINISH, DestroyVDSCommand, return: , log id: 389ad865 2021-11-17 21:12:54,596-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b'(ns8) was unexpectedly detected as 'Down' on VDS '10491335-e2e1-49f2-96c3-79331535542b'(ravn-kvm-9.ravnalaska.net) (expected on '6ee16602-c686-471a-9f65-e5952b813672') 2021-11-17 21:12:54,596-09 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] Migration of VM 'ns8' to host 'ravn-kvm-9.ravnalaska.net' failed: VM destroyed during the startup. 2021-11-17 21:12:54,642-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [] VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b'(ns8) moved from 'MigratingFrom' --> 'Up' 2021-11-17 21:12:54,642-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [] Adding VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b'(ns8) to re-run list 2021-11-17 21:12:54,644-09 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-5) [] Rerun VM 'fa1a2d6b-99cb-42bd-a343-91f314d5f47b'. Called from VDS 'ravn-kvm-8.ravnalaska.net' 2021-11-17 21:12:54,679-09 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-7232) [] START, MigrateStatusVDSCommand(HostName = ravn-kvm-8.ravnalaska.net, MigrateStatusVDSCommandParameters:{hostId='6ee16602-c686-471a-9f65-e5952b813672', vmId='fa1a2d6b-99cb-42bd-a343-91f314d5f47b'}), log id: 7ce5593e 2021-11-17 21:12:54,681-09 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-7232) [] FINISH, MigrateStatusVDSCommand, return: , log id: 7ce5593e 2021-11-17 21:12:54,695-09 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-7232) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: ns8, Source: ravn-kvm-8.ravnalaska.net, Destination: ravn-kvm-9.ravnalaska.net) 2021-11-17 21:12:54,697-09 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (EE-ManagedThreadFactory-engine-Thread-7232) [] Lock freed to object 'EngineLock:{exclusiveLocks='[fa1a2d6b-99cb-42bd-a343-91f314d5f47b=VM]', sharedLocks=''}' 2021-11-17 21:13:04,061-09 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-94) [] Fetched 1 VMs from VDS '10491335-e2e1-49f2-96c3-79331535542b' _______________________________ Gary Pedretty Director of IT Ravn Alaska Office: 907-266-8451 Mobile: 907-388-2247 Email: gary.pedretty(a)ravnalaska.com

6 7

Add static route to ovirt nodes
by Michael Thomas 20 Nov '21

20 Nov '21

I'm running ovirt 4.4.2 on CentOS 8.2. My ovirt nodes have two network addresses, ovirtmgmt and a second used for normal routed traffic to the cluster and WAN. After the ovirt nodes were set up, I found that I needed to add an extra static route to the cluster interface to allow the hosts to see my ceph storage nodes (to make the rbd images visible to the VMs): 10.9.0.0/16 via 10.13.0.1 I can add this route using three different methods: 1) ip route add 10.9.0.0/16 via 10.13.0.1 2) nmcli conn modify enp65s0f0 ipv4.routes "10.9.0.0/16 10.13.0.1" nmcli conn down enp65s0f0 nmcli conn up enp65s0f0 3) vi /etc/sysconfig/network-scripts/route-enp65s0f0 ifdown enp65s0f0 ifup enp65s0f0 However, when I reboot the host, the static route goes away. Methods 2 and 3 have always given me a persistent static route on other EL8 hosts, but not on my ovirt nodes. What is the correct way to add a persistent static route on an ovirt host? --Mike

2 1

CentOS 8 streams: Hosted engine deploy failed.
by Gilboa Davara 20 Nov '21

20 Nov '21

Hello all, I'm trying to redeploy one of the HE/Gluster clusters after a botched upgrade to Stream + simultaneous 2 x UPS failure that killed the previous setup. Post crash, the 3 nodes were cleaned up (manually + ovirt-hosted-engine-cleanup) and upgraded to Streams + reboot. I've made 3 attempts to deploy the HE, all failed with the same error. As far as I can see, everything works up until the final deployment, there the host fails to access the HE static address. Note: All machines have a static IP, resolved via local hosts file. Please advise (log attached). - Gilboa

1 2

oVirt 4.4.3 - some hosts in unassigned state - Get Host Capabilities failed: PKIX path building failed
by Roberto Nunin 19 Nov '21

19 Nov '21

Hi all We have an (old) installation with two DC in two different locations. Hosts where hosted engine is running are regularly reported UP (DC 1) Host into the other DC (connected by WAN lines) are reported as Unassigned (DC 2) Connection between DC is working. In events we can find lot of errors like: VDSM itmilu0xx-mng.example.com command Get Host Capabilities failed: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target We are NOT using a thirdy party SSL certificate. In engine.log these are recurring errors: 2021-11-16 10:28:49,370+01 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target 2021-11-16 10:28:49,372+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-100) [] Unable to RefreshCapabilities: VDSNetworkException: VDSGenericException: VDSNetworkException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Thanks in advance for any suggestion Roberto Nunin

1 1

Import from Vcenter failed with qemu-kvm: curl: The requested URL returned error: 401 Unauthorized
by roanne.philip＠voxtelecom.co.za 19 Nov '21

19 Nov '21

Hi I have ovirt HE Ver 4.4.7 with hosts on 4.3.10 I am trying to migrate vm's from vmware to ovirt but it fails. weirdly small vm imports successful and the larger ones fails have anyone experienced this issue qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized [ 2140.296725] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 2140.298168] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] [ 2140.299204] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated [ 2140.300183] sd 2:0:0:0: [sda] CDB: Read(10) 28 00 0a cc 9a 78 00 00 08 00 [ 2140.301291] blk_update_request: I/O error, dev sda, sector 181181048 qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized [ 2140.411642] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 2140.413110] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] [ 2140.414207] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated [ 2140.415198] sd 2:0:0:0: [sda] CDB: Read(10) 28 00 0a cc 9a 78 00 00 08 00 [ 2140.416326] blk_update_request: I/O error, dev sda, sector 181181048 qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized [ 2140.547845] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 2140.549289] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] [ 2140.550324] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated [ 2140.551296] sd 2:0:0:0: [sda] CDB: Read(10) 28 00 0a cc 9a 60 00 00 08 00 [ 2140.552397] blk_update_request: I/O error, dev sda, sector 181181024 qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized [ 2140.671078] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s [ 2140.672537] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] [ 2140.673581] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated [ 2140.674575] sd 2:0:0:0: [sda] CDB: Read(10) 28 00 0a cc 9a 60 00 00 08 00 [ 2140.675681] blk_update_request: I/O error, dev sda, sector 181181024 qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized qemu-kvm: curl: The requested URL returned error: 401 Unauthorized

1 0

Re: hosted-engine deploy - Network error during communication with the Host. (NFS)
by Matyi Szabolcs 19 Nov '21

19 Nov '21

*NFS sharing is fine I can mount from VM and also from localhost..*

1 0

How to renew vmconsole-proxy* certificates
by capelle＠labri.fr 19 Nov '21

19 Nov '21

Hi, Since a few weeks, we are not able to connect to the vmconsole proxy: $ ssh -t -p 2222 ovirt-vmconsole@ovirt ovirt-vmconsole@ovirt: Permission denied (publickey). Last successful login record: Mar 29 11:31:32 First login failure record: Mar 31 17:28:51 We tracked the issue to the following log in /var/log/ovirt-engine/engine.log: ERROR [org.ovirt.engine.core.services.VMConsoleProxyServlet] (default task-11) [] Error validating ticket: : sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Indeed, certificate /etc/pki/ovirt-engine/certs/vmconsole-proxy-helper.cer and others did expire: -- # grep 'Not After' /etc/pki/ovirt-engine/certs/vmconsole-proxy-* /etc/pki/ovirt-engine/certs/vmconsole-proxy-helper.cer: Not After : Mar 31 13:18:44 2021 GMT /etc/pki/ovirt-engine/certs/vmconsole-proxy-host.cer: Not After : Mar 31 13:18:44 2021 GMT /etc/pki/ovirt-engine/certs/vmconsole-proxy-user.cer: Not After : Mar 31 13:18:44 2021 GMT -- But we did not manage to found how to renew them. Any advice ? -- Benoît

3 2

Re: [PKI help] Renew vmconsole-proxy-helper.cer
by Richard Chan 18 Nov '21

18 Nov '21

running engine-setup should give you the opportunity to update expired or expiring certificates. > > If you don't want to upgrade your system you can run it with the --offline > option. > > Thank you! Richard Chan

1 0

Cannot to update hosts, nothing provides libvirt-daemon-kvm >= 7.6.0-2 needed by vdsm-4.40.90.4-1.el8.x86_64
by Alexandr Mikhailov 18 Nov '21

18 Nov '21

Hi! Last time i has updated my host running ovirt 4.4 in september. Everything was normally. After now in november i trying to update via UI but fave error that update failed. Checking over "dnf update" shows me that error in not resolved dependencies, and main is: nothing provides libvirt-daemon-kvm >= 7.6.0-2 needed by vdsm-4.40.90.3-1.el8.x86_64 nothing provides libvirt-daemon-kvm >= 7.6.0-2 needed by vdsm-4.40.90.4-1.el8.x86_64 Hosts running Centos 8.4. I already try to use one with stream - same trouble. My opinion this is problem with libvirt-daemon-kvm package in centos repo. Maybe i mistakes. Please help.

6 5

[PKI help] Renew vmconsole-proxy-helper.cer
by Richard Chan 18 Nov '21

18 Nov '21

This single cert has expired; how would I renew just this one certificate? -- Richard Chan

2 1