Migration failed due to an Error: Fatal error during migration
I am able to power on vms on newly upgraded host, But not able to migrate VMs from other hosts to new host or newly upgraded hosts to other hosts. This was worked fine before upgraded. Host logs ============================================================================== Unable to read from monitor: Connection reset by peer internal error: qemu unexpectedly closed the monitor: 2022-01-24T17:51:46.598571Z qemu-kvm: get_pci_config_device: Bad config > 2022-01-24T17:51:46.598627Z qemu-kvm: Failed to load PCIDevice:config 2022-01-24T17:51:46.598635Z qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj 2022-01-24T17:51:46.598642Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0/pcie-root-port' 2022-01-24T17:51:46.598830Z qemu-kvm: load of migration failed: Invalid argument Guest agent is not responding: QEMU guest agent is not connected Guest agent is not responding: QEMU guest agent is not connected Guest agent is not responding: QEMU guest agent is not connected Engine Logs =========================================================== 2022-01-24 11:31:25,080-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] Adding VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) to re-run list 2022-01-24 11:31:25,099-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2331914) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com). 2022-01-24 18:39:47,897-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-771) [a78e85d4-068a-41c1-a8fa-b3acd8c69317] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com, User: k.gunasekhar@non.keysight.com@KEYSIGHT). 2022-01-24 18:40:01,417-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) was unexpectedly detected as 'Down' on VDS 'ee23b44d-976d-4889-8769-59b56e4b23c0'(lcoskvmp03.cos.is.keysight.com) (expected on '0d58953f-b3cc-4bac-b3b2-08baeeee1bca') 2022-01-24 18:40:01,589-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) was unexpectedly detected as 'Down' on VDS 'ee23b44d-976d-4889-8769-59b56e4b23c0'(lcoskvmp03.cos.is.keysight.com) (expected on '0d58953f-b3cc-4bac-b3b2-08baeeee1bca') 2022-01-24 18:40:01,589-07 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'zzz2019' to host 'lcoskvmp03.cos.is.keysight.com' failed: VM destroyed during the startup. 2022-01-24 18:40:01,591-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-17) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) moved from 'MigratingFrom' --> 'Up' 2022-01-24 18:40:01,591-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-17) [] Adding VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) to re-run list 2022-01-24 18:40:01,611-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2348837) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com). [root@lcosovirt02 ovirt-engine]#
Hi, can you please provide versions of the non-upgraded hosts, of the upgraded hosts (I'm assuming you are using oVirt Node, if not, please provide the reports form `sos report -o rpm `) and of the engine? Il giorno mar 25 gen 2022 alle ore 02:49 Gunasekhar Kothapalli via Users < users@ovirt.org> ha scritto:
I am able to power on vms on newly upgraded host, But not able to migrate VMs from other hosts to new host or newly upgraded hosts to other hosts. This was worked fine before upgraded.
Host logs
============================================================================== Unable to read from monitor: Connection reset by peer internal error: qemu unexpectedly closed the monitor: 2022-01-24T17:51:46.598571Z qemu-kvm: get_pci_config_device: Bad config > 2022-01-24T17:51:46.598627Z qemu-kvm: Failed to load PCIDevice:config 2022-01-24T17:51:46.598635Z qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj 2022-01-24T17:51:46.598642Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0/pcie-root-port' 2022-01-24T17:51:46.598830Z qemu-kvm: load of migration failed: Invalid argument Guest agent is not responding: QEMU guest agent is not connected Guest agent is not responding: QEMU guest agent is not connected Guest agent is not responding: QEMU guest agent is not connected
Engine Logs ===========================================================
2022-01-24 11:31:25,080-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] Adding VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) to re-run list 2022-01-24 11:31:25,099-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2331914) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com). 2022-01-24 18:39:47,897-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-771) [a78e85d4-068a-41c1-a8fa-b3acd8c69317] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com, User: k.gunasekhar@non.keysight.com @KEYSIGHT). 2022-01-24 18:40:01,417-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) was unexpectedly detected as 'Down' on VDS 'ee23b44d-976d-4889-8769-59b56e4b23c0'( lcoskvmp03.cos.is.keysight.com) (expected on '0d58953f-b3cc-4bac-b3b2-08baeeee1bca') 2022-01-24 18:40:01,589-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) was unexpectedly detected as 'Down' on VDS 'ee23b44d-976d-4889-8769-59b56e4b23c0'( lcoskvmp03.cos.is.keysight.com) (expected on '0d58953f-b3cc-4bac-b3b2-08baeeee1bca') 2022-01-24 18:40:01,589-07 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'zzz2019' to host ' lcoskvmp03.cos.is.keysight.com' failed: VM destroyed during the startup. 2022-01-24 18:40:01,591-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-17) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) moved from 'MigratingFrom' --> 'Up' 2022-01-24 18:40:01,591-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-17) [] Adding VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) to re-run list 2022-01-24 18:40:01,611-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2348837) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com). [root@lcosovirt02 ovirt-engine]# _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2W5OEBPZM3TLS3...
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
On Tue, Jan 25, 2022 at 10:36 AM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Hi, can you please provide versions of the non-upgraded hosts, of the upgraded hosts (I'm assuming you are using oVirt Node, if not, please provide the reports form `sos report -o rpm `) and of the engine?
Il giorno mar 25 gen 2022 alle ore 02:49 Gunasekhar Kothapalli via Users < users@ovirt.org> ha scritto:
I am able to power on vms on newly upgraded host, But not able to migrate VMs from other hosts to new host or newly upgraded hosts to other hosts. This was worked fine before upgraded.
Host logs
============================================================================== Unable to read from monitor: Connection reset by peer internal error: qemu unexpectedly closed the monitor: 2022-01-24T17:51:46.598571Z qemu-kvm: get_pci_config_device: Bad config > 2022-01-24T17:51:46.598627Z qemu-kvm: Failed to load PCIDevice:config 2022-01-24T17:51:46.598635Z qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj 2022-01-24T17:51:46.598642Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0/pcie-root-port' 2022-01-24T17:51:46.598830Z qemu-kvm: load of migration failed: Invalid argument Guest agent is not responding: QEMU guest agent is not connected Guest agent is not responding: QEMU guest agent is not connected Guest agent is not responding: QEMU guest agent is not connected
Engine Logs ===========================================================
2022-01-24 11:31:25,080-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-21) [] Adding VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) to re-run list 2022-01-24 11:31:25,099-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2331914) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com). 2022-01-24 18:39:47,897-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-771) [a78e85d4-068a-41c1-a8fa-b3acd8c69317] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com, User: k.gunasekhar@non.keysight.com @KEYSIGHT). 2022-01-24 18:40:01,417-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) was unexpectedly detected as 'Down' on VDS 'ee23b44d-976d-4889-8769-59b56e4b23c0'( lcoskvmp03.cos.is.keysight.com) (expected on '0d58953f-b3cc-4bac-b3b2-08baeeee1bca') 2022-01-24 18:40:01,589-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) was unexpectedly detected as 'Down' on VDS 'ee23b44d-976d-4889-8769-59b56e4b23c0'( lcoskvmp03.cos.is.keysight.com) (expected on '0d58953f-b3cc-4bac-b3b2-08baeeee1bca') 2022-01-24 18:40:01,589-07 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-27) [] Migration of VM 'zzz2019' to host ' lcoskvmp03.cos.is.keysight.com' failed: VM destroyed during the startup. 2022-01-24 18:40:01,591-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-17) [] VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) moved from 'MigratingFrom' --> 'Up' 2022-01-24 18:40:01,591-07 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-17) [] Adding VM '9838c44b-710f-407a-b775-56bb0a3d4221'(zzz2019) to re-run list 2022-01-24 18:40:01,611-07 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-2348837) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Fatal error during migration (VM: zzz2019, Source: lcoskvmp07.cos.is.keysight.com, Destination: lcoskvmp03.cos.is.keysight.com). [root@lcosovirt02 ovirt-engine]# _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2W5OEBPZM3TLS3...
Hi, I answered on the other thread. This is a known problem with qemu-kvm 6.1. Please try to downgrade to 6.0. Regards, Liran
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SGXRHWYSGCRAZM...
This helped to resolve the issue. thank you
participants (3)
-
Gunasekhar Kothapalli -
Liran Rotenberg -
Sandro Bonazzola