I am trying to migrate vm from one host to another. I have already migrated 10 vm-s from that host to another, but 2 of them are having problems. In the dashboard event logs i get this: Migration failed due to an Error: Migration canceled (VM: Virtual-NS, Source: node2.ovirt.cluster.com, Destination: node3.ovirt.cluster.com). Engine log: 2024-04-11 11:11:21,490+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-74) [] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the refresh until migration is done 2024-04-11 11:11:35,446+02 WARN [org.ovirt.engine.core.utils.virtiowin.VirtioWinReader] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-75) [] Directory '/usr/share/virtio-win' doesn't exist. 2024-04-11 11:11:36,521+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the refresh until migration is done 2024-04-11 11:11:51,563+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the refresh until migration is done 2024-04-11 11:12:06,592+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-45) [] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the refresh until migration is done 2024-04-11 11:12:21,625+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-8) [] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the refresh until migration is done 2024-04-11 11:12:36,657+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-66) [] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the refresh until migration is done 2024-04-11 11:12:43,536+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [681c3996] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' was reported as Down on VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) 2024-04-11 11:12:43,536+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [681c3996] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) was unexpectedly detected as 'Down' on VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) (expected on 'c1f1069d-ed61-4ade-afc2-e6f649039386') 2024-04-11 11:12:43,536+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-5) [681c3996] START, DestroyVDSCommand(HostName = node3.ovirt.cluster.com, DestroyVmVDSCommandParameters:{hostId='d90dced4-6715-41b6-953c-119c4133f9db', vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 381766c1 2024-04-11 11:12:43,836+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-5) [681c3996] Failed to destroy VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' because VM does not exist, ignoring 2024-04-11 11:12:43,836+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-5) [681c3996] FINISH, DestroyVDSCommand, return: , log id: 381766c1 2024-04-11 11:12:43,836+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [681c3996] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) was unexpectedly detected as 'Down' on VDS 'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) (expected on 'c1f1069d-ed61-4ade-afc2-e6f649039386') 2024-04-11 11:12:43,836+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-5) [681c3996] Migration of VM 'Virtual-NS' to host 'node3.ovirt.cluster.com' failed: VM destroyed during the startup. 2024-04-11 11:12:43,842+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [681c3996] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) moved from 'MigratingFrom' --> 'Up' 2024-04-11 11:12:43,842+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-1) [681c3996] Adding VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) to re-run list 2024-04-11 11:12:43,843+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-1) [681c3996] Rerun VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'. Called from VDS 'node2.ovirt.cluster.com' 2024-04-11 11:12:43,889+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] START, MigrateStatusVDSCommand(HostName = node2.ovirt.cluster.com, MigrateStatusVDSCommandParameters:{hostId='c1f1069d-ed61-4ade-afc2-e6f649039386', vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d'}), log id: 41a9e7a3 2024-04-11 11:12:43,892+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] FINISH, MigrateStatusVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusReturn@4b4bced2, log id: 41a9e7a3 2024-04-11 11:12:43,920+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Migration canceled (VM: Virtual-NS, Source: node2.ovirt.cluster.com, Destination: node3.ovirt.cluster.com). 2024-04-11 11:12:43,922+02 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] Lock freed to object 'EngineLock:{exclusiveLocks='[c8e6aafe-1463-4db6-9d3b-76b234f9487d=VM]', sharedLocks=''}' 2024-04-11 11:12:51,685+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [] Fetched 13 VMs from VDS 'd90dced4-6715-41b6-953c-119c4133f9db' vdsm logs node3: 2024-04-11 11:12:42,360+0200 INFO (jsonrpc/3) [api.host] START getAllVmIoTunePolicies() from=::1,44562 (api:48) 2024-04-11 11:12:42,360+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'}, 'io_tune_policies_dict': {'1a2808ce-5ba4-4f19-9809-baffecb51d83': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/8766b16f-f587-4475-8e72-fc83486516b9/3616942e-88f1-4327-aad8-2d8d4550f8e7', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, 'f72cc0b9-0bdb-4d23-b50b-99e775d87f1c': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/f056a516-2257-4f0f-a972-031cf67dc4df/433fd71e-d4bb-4341-ad45-34fa4fd401a1', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD /aa2fa425-8bf3-44b6-b353-81de4e82764a/images/d5780d31-929b-46a6-b66e-38bcf57695d2/6b477529-daca-4480-9266-27d46ea6063e', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '148118a7-6b05-4d74-9d68-672d8c40f431': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/735770ac-7e7f-44c9-a768-3eeeac1ff0a8/aa387d94-ef62-45e9-aa11-e1ac6fcb138f', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, 'db853022-cee8-4f71-ad64-e70bb314686c': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/1294d8a3-ad40-4b06-9188-d59f527b921d/f944a7af-cda0-4d75-9c39-7b5063dae0cb', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/eed13b69-80cd-4742-b706-bce2f1370fa3/2163d081-bbe0-4d62-9d96-6bf0afd3260c', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '4490de5d-1a95-4906-a197-a471d95ab6b4': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/454c4196-d772-43fd-99bb-4ac90150cd3c/9779ba55-160d-431a-b408-857e106e5cbe', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/8d782dca-e45b-440c-b0a8-808d2f528dcd/70e4a01f-619e-4764-928f-1cc1207767e8', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec ': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '896840ab-2996-4c5b-afb2-c6a359ab6eda': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/454c4196-d772-43fd-99bb-4ac90150cd3c/9779ba55-160d-431a-b408-857e106e5cbe', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/a0defce2-7a1a-4872-8141-11df3e1bcba8/5f1745e6-3e0b-4997-915b-f13314fe9092', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '4eeb0711-a8e8-4c0d-aa29-0c9361e15554': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/260444be-9bd3-43be-bd08-9c494d2cc269/f5658dde- 9853-4d02-9b41-93501633e9e2', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/f8cb52ef-0dee-440c-96c7-9355522c8674/927d5b15-f46e-41a8-bf6e-053a6cc7cac8', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '0b528685-134b-48ff-9eb0-a8da6d1f3e70': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/ae598215-87ed-4ff3-9c3e-66a64e2f55da/320b1540-59c2-495e-96c3-3f523834662f', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '4e621664-b22b-430d-9e90-4f65ac4e99b7': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mn t/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/9e37bac4-10f3-4b6d-97db-fb2339b7282d/5595abba-8a8e-4dfe-a4c1-792d61317436', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '88516219-60bf-4d2f-8d50-96414327fe19': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/454c4196-d772-43fd-99bb-4ac90150cd3c/9779ba55-160d-431a-b408-857e106e5cbe', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/bb87b088-9c6f-4c6e-a2c5-4e10152ae10f/066bcd99-d1d6-411a-a0a2-b62a590112cb', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '281b503c-a76b-4755 -89f1-8a4b3eedb2a8': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/183bdbcc-b2af-4639-8866-c7066a211378/1ca39c83-59e7-4ed7-b5b9-11e5f3b42cb7', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sdb', 'path': '/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/7c15963f-ebc0-4c87-bc80-f8a64ef8f6db/6e3a4985-4cc9-4fa0-aa67-04297d31a107', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, '5d0fabc9-8671-4f81-9449-32e36091f4f1': {'policy': [], 'current_values': [{'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/460e3282-969f-48fc-9028-8e33e3bbd2dc/396a7e79-2cd5-4815-904a-cfa220a81b14', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_ bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sdb', 'path': '/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/e901a7d6-b029-4d83-93b2-522d1e061cd7/64749e34-56be-49fd-890b-496b19bdf45a', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, 'cc7931ff-8124-4724-9242-abea2ab5bf42': {'policy': [], 'current_values': [{'name': 'vda', 'path': '/run/vdsm/storage/73e03870-1bad-42d2-bf7b-a55041405c71/737c72eb-4e5b-46b6-ba7d-36f25880ceae/e0f8577d-a4bb-4a28-9780-a4cd1e6a2bb1', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}}} from=::1,44562 (api:54) 2024-04-11 11:12:42,752+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::1,44558 (api:48) 2024-04-11 11:12:42,773+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,44558, task_id=dce62f2b-b9e4-4641-969a-a34c4dccc5b7 (api:48) 2024-04-11 11:12:42,773+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '0.4', 'delay': '0.000354071', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '1.2', 'delay': '0.000259322', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '1.2', 'delay': '0.000269234', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '1.2', 'delay': '0.000352778', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '1.3', 'delay': '0.000313396', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '1.2', 'delay': '0.000272123', 'valid': True, 'ver sion': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '1.8', 'delay': '0.000403821', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,44558, task_id=dce62f2b-b9e4-4641-969a-a34c4dccc5b7 (api:54) 2024-04-11 11:12:42,774+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,44558, task_id=845b5ad7-5f99-40ac-b5b1-dd9700e7eca7 (api:48) 2024-04-11 11:12:42,774+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,44558, task_id=845b5ad7-5f99-40ac-b5b1-dd9700e7eca7 (api:54) 2024-04-11 11:12:42,788+0200 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,44558 (api:54) 2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') underlying process disconnected (vm:1135) 2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Release VM resources (vm:5357) 2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection (guestagent:438) 2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [vdsm.api] START teardownImage(sdUUID='8ab48aab-a522-4331-ba46-904e13f64d38', spUUID='07e52ac2-ab5f-11ec-b531-00163e400079', imgUUID='83f144ac-dfaa-4ede-a0f5-50338f6e1335', volUUID=None) from=internal, task_id=1ac73c98-9313-4e33-aee8-beaea54cb3a5 (api:48) 2024-04-11 11:12:43,518+0200 INFO (libvirt/events) [storage.StorageDomain] Removing image run directory '/run/vdsm/storage/8ab48aab-a522-4331-ba46-904e13f64d38/83f144ac-dfaa-4ede-a0f5-50338f6e1335' (blockSD:1387) 2024-04-11 11:12:43,518+0200 INFO (libvirt/events) [storage.fileUtils] Removing directory: /run/vdsm/storage/8ab48aab-a522-4331-ba46-904e13f64d38/83f144ac-dfaa-4ede-a0f5-50338f6e1335 (fileUtils:182) 2024-04-11 11:12:43,527+0200 INFO (jsonrpc/0) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.3.8.12,42396, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:48) 2024-04-11 11:12:43,532+0200 WARN (vm/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Couldn't destroy incoming VM: Domain not found: no domain with matching uuid 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' (vm:4100) 2024-04-11 11:12:43,532+0200 INFO (vm/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Changed state to Down: VM destroyed during the startup (code=10) (vm:1921) 2024-04-11 11:12:43,533+0200 INFO (vm/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection (guestagent:438) 2024-04-11 11:12:43,537+0200 INFO (jsonrpc/6) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:10.3.8.30,56070, flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:48) 2024-04-11 11:12:43,615+0200 INFO (libvirt/events) [storage.LVM] Deactivating lvs: vg=8ab48aab-a522-4331-ba46-904e13f64d38 lvs=['da4b0e32-9649-4f7d-834e-e0676a00916b'] (lvm:1802) 2024-04-11 11:12:43,728+0200 INFO (libvirt/events) [vdsm.api] FINISH teardownImage return=None from=internal, task_id=1ac73c98-9313-4e33-aee8-beaea54cb3a5 (api:54) 2024-04-11 11:12:43,729+0200 INFO (libvirt/events) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection (guestagent:438) 2024-04-11 11:12:43,729+0200 WARN (libvirt/events) [root] Attempting to remove a non existing net user: ovirtmgmt/c8e6aafe-1463-4db6-9d3b-76b234f9487d (libvirtnetwork:207) 2024-04-11 11:12:43,729+0200 INFO (libvirt/events) [vdsm.api] START inappropriateDevices(thiefId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') from=internal, task_id=834a0e43-879e-4d32-b719-50f4214b7d38 (api:48) 2024-04-11 11:12:43,730+0200 INFO (libvirt/events) [vdsm.api] FINISH inappropriateDevices return=None from=internal, task_id=834a0e43-879e-4d32-b719-50f4214b7d38 (api:54) 2024-04-11 11:12:43,832+0200 INFO (libvirt/events) [root] /usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err=b'' (hooks:122) 2024-04-11 11:12:43,833+0200 WARN (libvirt/events) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') trying to set state to Down when already Down (vm:702) 2024-04-11 11:12:43,833+0200 INFO (libvirt/events) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection (guestagent:438) 2024-04-11 11:12:43,834+0200 INFO (jsonrpc/0) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Can't undefine disconnected VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' (vm:2564) 2024-04-11 11:12:43,834+0200 INFO (jsonrpc/0) [api.virt] FINISH destroy return={'status': {'code': 0, 'message': 'Machine destroyed'}} from=::ffff:10.3.8.12,42396, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:54) 2024-04-11 11:12:43,835+0200 INFO (jsonrpc/6) [api] FINISH destroy error=Virtual machine does not exist: {'vmId': 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'} (api:129) 2024-04-11 11:12:43,835+0200 INFO (jsonrpc/6) [api.virt] FINISH destroy return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'}"}} from=::ffff:10.3.8.30,56070, flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:54) 2024-04-11 11:12:43,835+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.destroy failed (error 1) in 0.30 seconds (__init__:312) 2024-04-11 11:12:45,602+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::ffff:10.3.8.30,56070 (api:48) vdsm node2 log: 2024-04-11 11:12:27,504+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::1,33628 (api:48) 2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,33628, task_id=782990af-56f0-413f-b511-64e60e606537 (api:48) 2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '6.9', 'delay': '0.000313373', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00024877', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '3.1', 'delay': '0.000317606', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '2.4', 'delay': '0.000527974', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '2.4', 'delay': '0.000303832', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00288366', 'valid': True, 'versi on': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000218979', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=782990af-56f0-413f-b511-64e60e606537 (api:54) 2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,33628, task_id=3427f545-0183-49fd-98de-54a52823db1f (api:48) 2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,33628, task_id=3427f545-0183-49fd-98de-54a52823db1f (api:54) 2024-04-11 11:12:27,522+0200 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,33628 (api:54) 2024-04-11 11:12:30,597+0200 INFO (jsonrpc/4) [api.host] START getStats() from=::ffff:10.3.8.30,52426 (api:48) 2024-04-11 11:12:30,610+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::ffff:10.3.8.30,52426, task_id=eb4be2c4-919e-46d9-80c5-fb4bd111dc1f (api:48) 2024-04-11 11:12:30,610+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '0.0', 'delay': '0.000296656', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '5.5', 'delay': '0.00024877', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '6.2', 'delay': '0.000317606', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '5.5', 'delay': '0.000527974', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '5.5', 'delay': '0.000303832', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '5.4', 'delay': '0.00288366', 'valid': True, 'versi on': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '5.4', 'delay': '0.000218979', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.3.8.30,52426, task_id=eb4be2c4-919e-46d9-80c5-fb4bd111dc1f (api:54) 2024-04-11 11:12:30,611+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::ffff:10.3.8.30,52426, task_id=33c103f8-5221-448c-8ef3-900af4a0ff1d (api:48) 2024-04-11 11:12:30,611+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.3.8.30,52426, task_id=33c103f8-5221-448c-8ef3-900af4a0ff1d (api:54) 2024-04-11 11:12:30,618+0200 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.3.8.30,52426 (api:54) 2024-04-11 11:12:32,056+0200 INFO (migmon/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 94 < than old value 99, discarded (migration:255) 2024-04-11 11:12:32,057+0200 WARN (migmon/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Migration stalling: remaining (479MiB) > lowmark (185MiB). (migration:873) 2024-04-11 11:12:32,057+0200 INFO (migmon/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Migration Progress: 470.127 seconds elapsed, 98% of data processed, total data: 16484MB, processed data: 28862MB, remaining data: 479MB, transfer speed 496Mbps, zero pages: 515507MB, compressed: 9MB, dirty rate: 13508, memory iteration: 9 (migration:886) 2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,33628, vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:48) 2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'} (api:129) 2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'}"}} from=::1,33628, vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:54) 2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2024-04-11 11:12:33,754+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628, task_id=16d2af05-faac-4854-89e6-fb3f39452318 (api:48) 2024-04-11 11:12:33,754+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '3.1', 'delay': '0.000296656', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=16d2af05-faac-4854-89e6-fb3f39452318 (api:54) 2024-04-11 11:12:34,124+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628, task_id=bdb6aa5e-b9c4-4832-bb8f-fec782cbbde0 (api:48) 2024-04-11 11:12:34,124+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '3.5', 'delay': '0.000296656', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=bdb6aa5e-b9c4-4832-bb8f-fec782cbbde0 (api:54) 2024-04-11 11:12:34,928+0200 INFO (jsonrpc/4) [vdsm.api] START getSpmStatus(spUUID='07e52ac2-ab5f-11ec-b531-00163e400079') from=::ffff:10.3.8.30,52426, task_id=5e808bd5-fefd-454c-a5f6-4785bfad013e (api:48) 2024-04-11 11:12:34,931+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::1,33628 (api:48) 2024-04-11 11:12:34,944+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,33628, task_id=2e5ceb8f-f518-46eb-88b7-e9caf55de687 (api:48) 2024-04-11 11:12:34,944+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '4.3', 'delay': '0.000296656', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '4.3', 'delay': '0.00024877', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '0.5', 'delay': '0.000262637', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '4.3', 'delay': '0.000527974', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '4.3', 'delay': '0.000303832', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '4.3', 'delay': '0.00288366', 'valid': True, 'versi on': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '4.3', 'delay': '0.000218979', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=2e5ceb8f-f518-46eb-88b7-e9caf55de687 (api:54) 2024-04-11 11:12:34,945+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,33628, task_id=23391530-d739-47ee-8799-d3db76e805ca (api:48) 2024-04-11 11:12:34,945+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,33628, task_id=23391530-d739-47ee-8799-d3db76e805ca (api:54) 2024-04-11 11:12:34,946+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus return={'spm_st': {'spmStatus': 'SPM', 'spmLver': 13, 'spmId': 2}} from=::ffff:10.3.8.30,52426, task_id=5e808bd5-fefd-454c-a5f6-4785bfad013e (api:54) 2024-04-11 11:12:34,952+0200 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,33628 (api:54) 2024-04-11 11:12:34,989+0200 INFO (jsonrpc/3) [vdsm.api] START getStoragePoolInfo(spUUID='07e52ac2-ab5f-11ec-b531-00163e400079') from=::ffff:10.3.8.30,52436, task_id=61162b84-6df1-4916-a959-20214aa7bc2b (api:48) 2024-04-11 11:12:34,992+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo return={'info': {'domains': '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0:Active,848dac7f-a73d-476f-9371-b54ac8b647de:Active,8ab48aab-a522-4331-ba46-904e13f64d38:Active,c0b21e2c-6b4c-4769-8244-aa69a32c27f2:Active,aa2fa425-8bf3-44b6-b353-81de4e82764a:Active,7f655239-96f9-48f9-99e0-b947220c289e:Active,73e03870-1bad-42d2-bf7b-a55041405c71:Active', 'isoprefix': '', 'lver': 13, 'master_uuid': 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2', 'master_ver': 2, 'name': 'No Description', 'pool_status': 'connected', 'spm_id': 2, 'type': 'FCP', 'version': '5'}, 'dominfo': {'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '3298132230144', 'diskfree': '1233997791232'}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '321719894016', 'diskfree': '316619620352'}, '8ab48aab-a522-4331-ba46-904e13f6 4d38': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '2198620602368', 'diskfree': '1906965479424'}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '3298132230144', 'diskfree': '1311709855744'}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '106971529216', 'diskfree': '63350767616'}, '7f655239-96f9-48f9-99e0-b947220c289e': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '1099108974592', 'diskfree': '1071460122624'}, '73e03870-1bad-42d2-bf7b-a55041405c71': {'status': 'Active', 'alerts': [], 'isoprefix': '', 'version': 5, 'disktotal': '106971529216', 'diskfree': '11945377792'}}} from=::ffff:10.3.8.30,52436, task_id=61162b84-6df1-4916-a959-20214aa7bc2b (api:54) 2024-04-11 11:12:35,536+0200 INFO (periodic/2) [vdsm.api] START getVolumeSize(sdUUID='8ab48aab-a522-4331-ba46-904e13f64d38', spUUID='07e52ac2-ab5f-11ec-b531-00163e400079', imgUUID='83f144ac-dfaa-4ede-a0f5-50338f6e1335', volUUID='da4b0e32-9649-4f7d-834e-e0676a00916b') from=internal, task_id=d6717cfc-a75c-4f1d-baa9-887711f4732f (api:48) 2024-04-11 11:12:35,536+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize return={'apparentsize': '110595407872', 'truesize': '110595407872'} from=internal, task_id=d6717cfc-a75c-4f1d-baa9-887711f4732f (api:54) 2024-04-11 11:12:35,537+0200 INFO (periodic/1) [vdsm.api] START getVolumeSize(sdUUID='c0b21e2c-6b4c-4769-8244-aa69a32c27f2', spUUID='07e52ac2-ab5f-11ec-b531-00163e400079', imgUUID='f194f2e8-0191-4073-90f6-21e6ddc3e350', volUUID='f66d26c5-7b48-487a-92d2-dba2d588abff') from=internal, task_id=f65d5fdc-0b1a-4cc0-9a82-8b0a4c28556b (api:48) 2024-04-11 11:12:35,537+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize return={'apparentsize': '340376158208', 'truesize': '340376158208'} from=internal, task_id=f65d5fdc-0b1a-4cc0-9a82-8b0a4c28556b (api:54) 2024-04-11 11:12:37,549+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::1,33628 (api:48) 2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,33628, task_id=bd232d01-4552-4937-b2a3-c4b8ec6d9910 (api:48) 2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '6.9', 'delay': '0.000296656', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '2.4', 'delay': '0.000202745', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '3.1', 'delay': '0.000262637', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '2.4', 'delay': '0.000343811', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '2.4', 'delay': '0.000232965', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00022068', 'valid': True, 'vers ion': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '2.4', 'delay': '0.000188189', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=bd232d01-4552-4937-b2a3-c4b8ec6d9910 (api:54) 2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,33628, task_id=a6903cbf-c31e-45d4-b662-5167cd499faf (api:48) 2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,33628, task_id=a6903cbf-c31e-45d4-b662-5167cd499faf (api:54) 2024-04-11 11:12:37,569+0200 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,33628 (api:54) 2024-04-11 11:12:40,651+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:10.3.8.30,52426 (api:48) 2024-04-11 11:12:40,651+0200 INFO (jsonrpc/0) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than old value 99, discarded (migration:255) 2024-04-11 11:12:40,653+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:10.3.8.30,52426 (api:54) 2024-04-11 11:12:40,867+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=f679c2f0-24e4-454c-b511-653b39c23998 (api:48) 2024-04-11 11:12:40,867+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '0.2', 'delay': '0.000529618', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '0.2', 'delay': '0.000202745', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '0.2', 'delay': '0.000262637', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '0.2', 'delay': '0.000343811', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '0.2', 'delay': '0.000232965', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00022068', 'valid': True, 'ver sion': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '0.2', 'delay': '0.000188189', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=f679c2f0-24e4-454c-b511-653b39c23998 (api:54) 2024-04-11 11:12:42,059+0200 INFO (migmon/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than old value 99, discarded (migration:255) 2024-04-11 11:12:42,059+0200 WARN (migmon/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Migration stalling: remaining (709MiB) > lowmark (185MiB). (migration:873) 2024-04-11 11:12:42,059+0200 WARN (migmon/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Aborting migration (migration:922) 2024-04-11 11:12:42,455+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,33630 (api:48) 2024-04-11 11:12:42,455+0200 INFO (jsonrpc/1) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than old value 99, discarded (migration:255) 2024-04-11 11:12:42,456+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,33630 (api:54) 2024-04-11 11:12:42,462+0200 INFO (jsonrpc/5) [api.host] START getAllVmIoTunePolicies() from=::1,33630 (api:48) 2024-04-11 11:12:42,462+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'}, 'io_tune_policies_dict': {'c8e6aafe-1463-4db6-9d3b-76b234f9487d': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/a57a0a15-1b0a-4f7d-83a0-d59815080caf/d0ba288d-23cf-451f-bd76-d4a7bde6032f', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/83f144ac-dfaa-4ede-a0f5-50338f6e1335/da4b0e32-9649-4f7d-834e-e0676a00916b', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}, 'b77300a2-2422-4bfb-a6f6-eea9f4bd4e89': {'policy': [], 'current_values': [{'name': 'sdc', 'path': '/rhev/data-center/mnt/blockSD /aa2fa425-8bf3-44b6-b353-81de4e82764a/images/1294d8a3-ad40-4b06-9188-d59f527b921d/f944a7af-cda0-4d75-9c39-7b5063dae0cb', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda', 'path': '/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/f194f2e8-0191-4073-90f6-21e6ddc3e350/f66d26c5-7b48-487a-92d2-dba2d588abff', 'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0, 'read_iops_sec': 0}}]}}} from=::1,33630 (api:54) 2024-04-11 11:12:43,518+0200 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,33628, vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:48) 2024-04-11 11:12:43,519+0200 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'} (api:129) 2024-04-11 11:12:43,519+0200 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'}"}} from=::1,33628, vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:54) 2024-04-11 11:12:43,519+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) 2024-04-11 11:12:43,523+0200 ERROR (migsrc/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') operation aborted: migration out job: canceled by client (migration:331) 2024-04-11 11:12:43,771+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628, task_id=678887b6-dc37-4ad3-b1e5-b6db22c9f0e4 (api:48) 2024-04-11 11:12:43,771+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '3.1', 'delay': '0.000529618', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=678887b6-dc37-4ad3-b1e5-b6db22c9f0e4 (api:54) 2024-04-11 11:12:43,839+0200 INFO (migsrc/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Switching from State.STARTED to State.FAILED (migration:234) 2024-04-11 11:12:43,839+0200 ERROR (migsrc/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Failed to migrate (migration:503) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 477, in _regular_run time.time(), machineParams File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 578, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 667, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 596, in _perform_migration self._migration_flags) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 159, in call return getattr(self._vm._dom, name)(*a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: operation aborted: migration out job: canceled by client 2024-04-11 11:12:43,839+0200 INFO (migsrc/c8e6aafe) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Enabling drive monitoring (drivemonitor:52) 2024-04-11 11:12:43,890+0200 INFO (jsonrpc/7) [api.virt] START getMigrationStatus() from=::ffff:10.3.8.30,52426, flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:48) 2024-04-11 11:12:43,890+0200 INFO (jsonrpc/7) [virt.vm] (vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than old value 99, discarded (migration:255) 2024-04-11 11:12:43,890+0200 INFO (jsonrpc/7) [api.virt] FINISH getMigrationStatus return={'status': {'code': 0, 'message': 'Done'}, 'migrationStats': {'status': {'code': 47, 'message': 'Migration canceled'}, 'progress': 99}} from=::ffff:10.3.8.30,52426, flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:54) 2024-04-11 11:12:44,215+0200 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628, task_id=f98771e0-235b-4c8e-a445-1f72ab61c1d8 (api:48) 2024-04-11 11:12:44,215+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '3.6', 'delay': '0.000529618', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=f98771e0-235b-4c8e-a445-1f72ab61c1d8 (api:54) 2024-04-11 11:12:44,976+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::1,33628 (api:48) 2024-04-11 11:12:44,989+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,33628, task_id=f7db90cd-d4ed-4670-83f1-d4336f07cc2e (api:48) 2024-04-11 11:12:44,989+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0, 'lastCheck': '4.4', 'delay': '0.000529618', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0, 'lastCheck': '4.4', 'delay': '0.000202745', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck': '0.5', 'delay': '0.000303342', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0, 'lastCheck': '4.3', 'delay': '0.000343811', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0, 'lastCheck': '4.3', 'delay': '0.000232965', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0, 'lastCheck': '4.4', 'delay': '0.00022068', 'valid': True, 'vers ion': 5, 'acquired': True, 'actual': True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck': '4.4', 'delay': '0.000188189', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,33628, task_id=f7db90cd-d4ed-4670-83f1-d4336f07cc2e (api:54) 2024-04-11 11:12:44,990+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,33628, task_id=6f2364c5-df82-4104-aed4-e0b71d58ac95 (api:48) 2024-04-11 11:12:44,990+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,33628, task_id=6f2364c5-df82-4104-aed4-e0b71d58ac95 (api:54)