4:23 a.m.
I am trying to migrate vm from one host to another.
I have already migrated 10 vm-s from that host to another, but 2 of them are having
problems.
In the dashboard event logs i get this:
Migration failed due to an Error: Migration canceled (VM: Virtual-NS, Source:
node2.ovirt.cluster.com, Destination: node3.ovirt.cluster.com).
Engine log:
2024-04-11 11:11:21,490+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-74) [] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the
refresh until migration is done
2024-04-11 11:11:35,446+02 WARN [org.ovirt.engine.core.utils.virtiowin.VirtioWinReader]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-75) [] Directory
'/usr/share/virtio-win' doesn't exist.
2024-04-11 11:11:36,521+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the
refresh until migration is done
2024-04-11 11:11:51,563+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-44) [] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the
refresh until migration is done
2024-04-11 11:12:06,592+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-45) [] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the
refresh until migration is done
2024-04-11 11:12:21,625+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-8) [] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the
refresh until migration is done
2024-04-11 11:12:36,657+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-66) [] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' is migrating to VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) ignoring it in the
refresh until migration is done
2024-04-11 11:12:43,536+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-5) [681c3996] VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' was
reported as Down on VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com)
2024-04-11 11:12:43,536+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-5) [681c3996] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) was unexpectedly detected as
'Down' on VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) (expected on
'c1f1069d-ed61-4ade-afc2-e6f649039386')
2024-04-11 11:12:43,536+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-5)
[681c3996] START, DestroyVDSCommand(HostName = node3.ovirt.cluster.com,
DestroyVmVDSCommandParameters:{hostId='d90dced4-6715-41b6-953c-119c4133f9db',
vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d', secondsToWait='0',
gracefully='false', reason='', ignoreNoVm='true'}), log id:
381766c1
2024-04-11 11:12:43,836+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-5)
[681c3996] Failed to destroy VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d' because VM
does not exist, ignoring
2024-04-11 11:12:43,836+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-5)
[681c3996] FINISH, DestroyVDSCommand, return: , log id: 381766c1
2024-04-11 11:12:43,836+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-5) [681c3996] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) was unexpectedly detected as
'Down' on VDS
'd90dced4-6715-41b6-953c-119c4133f9db'(node3.ovirt.cluster.com) (expected on
'c1f1069d-ed61-4ade-afc2-e6f649039386')
2024-04-11 11:12:43,836+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-5) [681c3996] Migration of VM 'Virtual-NS' to host
'node3.ovirt.cluster.com' failed: VM destroyed during the startup.
2024-04-11 11:12:43,842+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-1) [681c3996] VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) moved from
'MigratingFrom' --> 'Up'
2024-04-11 11:12:43,842+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-1) [681c3996] Adding VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d'(Virtual-NS) to re-run list
2024-04-11 11:12:43,843+02 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-1)
[681c3996] Rerun VM 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'. Called from VDS
'node2.ovirt.cluster.com'
2024-04-11 11:12:43,889+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] START,
MigrateStatusVDSCommand(HostName = node2.ovirt.cluster.com,
MigrateStatusVDSCommandParameters:{hostId='c1f1069d-ed61-4ade-afc2-e6f649039386',
vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d'}), log id: 41a9e7a3
2024-04-11 11:12:43,892+02 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] FINISH,
MigrateStatusVDSCommand, return:
org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusReturn@4b4bced2, log id: 41a9e7a3
2024-04-11 11:12:43,920+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed due to an Error: Migration canceled
(VM: Virtual-NS, Source: node2.ovirt.cluster.com, Destination: node3.ovirt.cluster.com).
2024-04-11 11:12:43,922+02 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
(EE-ManagedThreadFactory-engine-Thread-9915782) [681c3996] Lock freed to object
'EngineLock:{exclusiveLocks='[c8e6aafe-1463-4db6-9d3b-76b234f9487d=VM]',
sharedLocks=''}'
2024-04-11 11:12:51,685+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-55) [] Fetched 13 VMs
from VDS 'd90dced4-6715-41b6-953c-119c4133f9db'
vdsm logs node3:
2024-04-11 11:12:42,360+0200 INFO (jsonrpc/3) [api.host] START getAllVmIoTunePolicies()
from=::1,44562 (api:48)
2024-04-11 11:12:42,360+0200 INFO (jsonrpc/3) [api.host] FINISH getAllVmIoTunePolicies
return={'status': {'code': 0, 'message': 'Done'},
'io_tune_policies_dict': {'1a2808ce-5ba4-4f19-9809-baffecb51d83':
{'policy': [], 'current_values': [{'name': 'sda',
'path':
'/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/8766b16f-f587-4475-8e72-fc83486516b9/3616942e-88f1-4327-aad8-2d8d4550f8e7',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, 'f72cc0b9-0bdb-4d23-b50b-99e775d87f1c':
{'policy': [], 'current_values': [{'name': 'sda',
'path':
'/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/f056a516-2257-4f0f-a972-031cf67dc4df/433fd71e-d4bb-4341-ad45-34fa4fd401a1',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sdc', 'path':
'/rhev/data-center/mnt/blockSD
/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/d5780d31-929b-46a6-b66e-38bcf57695d2/6b477529-daca-4480-9266-27d46ea6063e',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '148118a7-6b05-4d74-9d68-672d8c40f431':
{'policy': [], 'current_values': [{'name': 'sda',
'path':
'/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/735770ac-7e7f-44c9-a768-3eeeac1ff0a8/aa387d94-ef62-45e9-aa11-e1ac6fcb138f',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, 'db853022-cee8-4f71-ad64-e70bb314686c':
{'policy': [], 'current_values': [{'name': 'sdc',
'path':
'/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/1294d8a3-ad40-4b06-9188-d59f527b921d/f944a7af-cda0-4d75-9c39-7b5063dae0cb',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec':
0, 'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name':
'sda', 'path':
'/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/eed13b69-80cd-4742-b706-bce2f1370fa3/2163d081-bbe0-4d62-9d96-6bf0afd3260c',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '4490de5d-1a95-4906-a197-a471d95ab6b4':
{'policy': [], 'current_values': [{'name': 'sdc',
'path':
'/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/454c4196-d772-43fd-99bb-4ac90150cd3c/9779ba55-160d-431a-b408-857e106e5cbe',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sda', 'path':
'/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/8d782dca-e45b-440c-b0a8-808d2f528dcd/70e4a01f-619e-4764-928f-1cc1207767e8',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec
': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '896840ab-2996-4c5b-afb2-c6a359ab6eda':
{'policy': [], 'current_values': [{'name': 'sdc',
'path':
'/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/454c4196-d772-43fd-99bb-4ac90150cd3c/9779ba55-160d-431a-b408-857e106e5cbe',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sda', 'path':
'/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/a0defce2-7a1a-4872-8141-11df3e1bcba8/5f1745e6-3e0b-4997-915b-f13314fe9092',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '4eeb0711-a8e8-4c0d-aa29-0c9361e15554':
{'policy': [], 'current_values': [{'name': 'sdc',
'path':
'/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/260444be-9bd3-43be-bd08-9c494d2cc269/f5658dde-
9853-4d02-9b41-93501633e9e2', 'ioTune': {'total_bytes_sec': 0,
'read_bytes_sec': 0, 'write_bytes_sec': 0, 'total_iops_sec': 0,
'write_iops_sec': 0, 'read_iops_sec': 0}}, {'name': 'sda',
'path':
'/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/f8cb52ef-0dee-440c-96c7-9355522c8674/927d5b15-f46e-41a8-bf6e-053a6cc7cac8',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '0b528685-134b-48ff-9eb0-a8da6d1f3e70':
{'policy': [], 'current_values': [{'name': 'sda',
'path':
'/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/ae598215-87ed-4ff3-9c3e-66a64e2f55da/320b1540-59c2-495e-96c3-3f523834662f',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '4e621664-b22b-430d-9e90-4f65ac4e99b7':
{'policy': [], 'current_values': [{'name': 'sda',
'path': '/rhev/data-center/mn
t/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/9e37bac4-10f3-4b6d-97db-fb2339b7282d/5595abba-8a8e-4dfe-a4c1-792d61317436',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '88516219-60bf-4d2f-8d50-96414327fe19':
{'policy': [], 'current_values': [{'name': 'sdc',
'path':
'/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/454c4196-d772-43fd-99bb-4ac90150cd3c/9779ba55-160d-431a-b408-857e106e5cbe',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sda', 'path':
'/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/bb87b088-9c6f-4c6e-a2c5-4e10152ae10f/066bcd99-d1d6-411a-a0a2-b62a590112cb',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '281b503c-a76b-4755
-89f1-8a4b3eedb2a8': {'policy': [], 'current_values':
[{'name': 'sda', 'path':
'/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/183bdbcc-b2af-4639-8866-c7066a211378/1ca39c83-59e7-4ed7-b5b9-11e5f3b42cb7',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sdb', 'path':
'/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/7c15963f-ebc0-4c87-bc80-f8a64ef8f6db/6e3a4985-4cc9-4fa0-aa67-04297d31a107',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, '5d0fabc9-8671-4f81-9449-32e36091f4f1':
{'policy': [], 'current_values': [{'name': 'sda',
'path':
'/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/460e3282-969f-48fc-9028-8e33e3bbd2dc/396a7e79-2cd5-4815-904a-cfa220a81b14',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0, 'write_
bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sdb', 'path':
'/rhev/data-center/mnt/blockSD/2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0/images/e901a7d6-b029-4d83-93b2-522d1e061cd7/64749e34-56be-49fd-890b-496b19bdf45a',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, 'cc7931ff-8124-4724-9242-abea2ab5bf42':
{'policy': [], 'current_values': [{'name': 'vda',
'path':
'/run/vdsm/storage/73e03870-1bad-42d2-bf7b-a55041405c71/737c72eb-4e5b-46b6-ba7d-36f25880ceae/e0f8577d-a4bb-4a28-9780-a4cd1e6a2bb1',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}}} from=::1,44562 (api:54)
2024-04-11 11:12:42,752+0200 INFO (jsonrpc/2) [api.host] START getStats() from=::1,44558
(api:48)
2024-04-11 11:12:42,773+0200 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=())
from=::1,44558, task_id=dce62f2b-b9e4-4641-969a-a34c4dccc5b7 (api:48)
2024-04-11 11:12:42,773+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '0.4', 'delay': '0.000354071',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0,
'lastCheck': '1.2', 'delay': '0.000259322',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '1.2', 'delay': '0.000269234',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '1.2', 'delay': '0.000352778',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '1.3', 'delay': '0.000313396',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '1.2', 'delay': '0.000272123',
'valid': True, 'ver
sion': 5, 'acquired': True, 'actual': True},
'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0, 'lastCheck':
'1.8', 'delay': '0.000403821', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}} from=::1,44558,
task_id=dce62f2b-b9e4-4641-969a-a34c4dccc5b7 (api:54)
2024-04-11 11:12:42,774+0200 INFO (jsonrpc/2) [vdsm.api] START multipath_health()
from=::1,44558, task_id=845b5ad7-5f99-40ac-b5b1-dd9700e7eca7 (api:48)
2024-04-11 11:12:42,774+0200 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health
return={} from=::1,44558, task_id=845b5ad7-5f99-40ac-b5b1-dd9700e7eca7 (api:54)
2024-04-11 11:12:42,788+0200 INFO (jsonrpc/2) [api.host] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'},
'info': (suppressed)} from=::1,44558 (api:54)
2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') underlying process disconnected
(vm:1135)
2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Release VM resources (vm:5357)
2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection
(guestagent:438)
2024-04-11 11:12:43,517+0200 INFO (libvirt/events) [vdsm.api] START
teardownImage(sdUUID='8ab48aab-a522-4331-ba46-904e13f64d38',
spUUID='07e52ac2-ab5f-11ec-b531-00163e400079',
imgUUID='83f144ac-dfaa-4ede-a0f5-50338f6e1335', volUUID=None) from=internal,
task_id=1ac73c98-9313-4e33-aee8-beaea54cb3a5 (api:48)
2024-04-11 11:12:43,518+0200 INFO (libvirt/events) [storage.StorageDomain] Removing image
run directory
'/run/vdsm/storage/8ab48aab-a522-4331-ba46-904e13f64d38/83f144ac-dfaa-4ede-a0f5-50338f6e1335'
(blockSD:1387)
2024-04-11 11:12:43,518+0200 INFO (libvirt/events) [storage.fileUtils] Removing
directory:
/run/vdsm/storage/8ab48aab-a522-4331-ba46-904e13f64d38/83f144ac-dfaa-4ede-a0f5-50338f6e1335
(fileUtils:182)
2024-04-11 11:12:43,527+0200 INFO (jsonrpc/0) [api.virt] START
destroy(gracefulAttempts=1) from=::ffff:10.3.8.12,42396,
vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:48)
2024-04-11 11:12:43,532+0200 WARN (vm/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Couldn't destroy incoming VM:
Domain not found: no domain with matching uuid
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' (vm:4100)
2024-04-11 11:12:43,532+0200 INFO (vm/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Changed state to Down: VM destroyed
during the startup (code=10) (vm:1921)
2024-04-11 11:12:43,533+0200 INFO (vm/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection
(guestagent:438)
2024-04-11 11:12:43,537+0200 INFO (jsonrpc/6) [api.virt] START
destroy(gracefulAttempts=1) from=::ffff:10.3.8.30,56070, flow_id=681c3996,
vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:48)
2024-04-11 11:12:43,615+0200 INFO (libvirt/events) [storage.LVM] Deactivating lvs:
vg=8ab48aab-a522-4331-ba46-904e13f64d38
lvs=['da4b0e32-9649-4f7d-834e-e0676a00916b'] (lvm:1802)
2024-04-11 11:12:43,728+0200 INFO (libvirt/events) [vdsm.api] FINISH teardownImage
return=None from=internal, task_id=1ac73c98-9313-4e33-aee8-beaea54cb3a5 (api:54)
2024-04-11 11:12:43,729+0200 INFO (libvirt/events) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection
(guestagent:438)
2024-04-11 11:12:43,729+0200 WARN (libvirt/events) [root] Attempting to remove a non
existing net user: ovirtmgmt/c8e6aafe-1463-4db6-9d3b-76b234f9487d (libvirtnetwork:207)
2024-04-11 11:12:43,729+0200 INFO (libvirt/events) [vdsm.api] START
inappropriateDevices(thiefId='c8e6aafe-1463-4db6-9d3b-76b234f9487d')
from=internal, task_id=834a0e43-879e-4d32-b719-50f4214b7d38 (api:48)
2024-04-11 11:12:43,730+0200 INFO (libvirt/events) [vdsm.api] FINISH inappropriateDevices
return=None from=internal, task_id=834a0e43-879e-4d32-b719-50f4214b7d38 (api:54)
2024-04-11 11:12:43,832+0200 INFO (libvirt/events) [root]
/usr/libexec/vdsm/hooks/after_vm_destroy/delete_vhostuserclient_hook: rc=0 err=b''
(hooks:122)
2024-04-11 11:12:43,833+0200 WARN (libvirt/events) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') trying to set state to Down when
already Down (vm:702)
2024-04-11 11:12:43,833+0200 INFO (libvirt/events) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Stopping connection
(guestagent:438)
2024-04-11 11:12:43,834+0200 INFO (jsonrpc/0) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Can't undefine disconnected VM
'c8e6aafe-1463-4db6-9d3b-76b234f9487d' (vm:2564)
2024-04-11 11:12:43,834+0200 INFO (jsonrpc/0) [api.virt] FINISH destroy
return={'status': {'code': 0, 'message': 'Machine
destroyed'}} from=::ffff:10.3.8.12,42396, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d
(api:54)
2024-04-11 11:12:43,835+0200 INFO (jsonrpc/6) [api] FINISH destroy error=Virtual machine
does not exist: {'vmId': 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'}
(api:129)
2024-04-11 11:12:43,835+0200 INFO (jsonrpc/6) [api.virt] FINISH destroy
return={'status': {'code': 1, 'message': "Virtual machine
does not exist: {'vmId': 'c8e6aafe-1463-4db6-9d3b-76b234f9487d'}"}}
from=::ffff:10.3.8.30,56070, flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d
(api:54)
2024-04-11 11:12:43,835+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.destroy
failed (error 1) in 0.30 seconds (__init__:312)
2024-04-11 11:12:45,602+0200 INFO (jsonrpc/1) [api.host] START getStats()
from=::ffff:10.3.8.30,56070 (api:48)
vdsm node2 log:
2024-04-11 11:12:27,504+0200 INFO (jsonrpc/1) [api.host] START getStats() from=::1,33628
(api:48)
2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=())
from=::1,33628, task_id=782990af-56f0-413f-b511-64e60e606537 (api:48)
2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '6.9', 'delay': '0.000313373',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '2.4', 'delay': '0.00024877',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0,
'lastCheck': '3.1', 'delay': '0.000317606',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '2.4', 'delay': '0.000527974',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '2.4', 'delay': '0.000303832',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '2.3', 'delay': '0.00288366',
'valid': True, 'versi
on': 5, 'acquired': True, 'actual': True},
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck':
'2.3', 'delay': '0.000218979', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}} from=::1,33628,
task_id=782990af-56f0-413f-b511-64e60e606537 (api:54)
2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] START multipath_health()
from=::1,33628, task_id=3427f545-0183-49fd-98de-54a52823db1f (api:48)
2024-04-11 11:12:27,516+0200 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health
return={} from=::1,33628, task_id=3427f545-0183-49fd-98de-54a52823db1f (api:54)
2024-04-11 11:12:27,522+0200 INFO (jsonrpc/1) [api.host] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'},
'info': (suppressed)} from=::1,33628 (api:54)
2024-04-11 11:12:30,597+0200 INFO (jsonrpc/4) [api.host] START getStats()
from=::ffff:10.3.8.30,52426 (api:48)
2024-04-11 11:12:30,610+0200 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=())
from=::ffff:10.3.8.30,52426, task_id=eb4be2c4-919e-46d9-80c5-fb4bd111dc1f (api:48)
2024-04-11 11:12:30,610+0200 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '0.0', 'delay': '0.000296656',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '5.5', 'delay': '0.00024877',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0,
'lastCheck': '6.2', 'delay': '0.000317606',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '5.5', 'delay': '0.000527974',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '5.5', 'delay': '0.000303832',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '5.4', 'delay': '0.00288366',
'valid': True, 'versi
on': 5, 'acquired': True, 'actual': True},
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck':
'5.4', 'delay': '0.000218979', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}}
from=::ffff:10.3.8.30,52426, task_id=eb4be2c4-919e-46d9-80c5-fb4bd111dc1f (api:54)
2024-04-11 11:12:30,611+0200 INFO (jsonrpc/4) [vdsm.api] START multipath_health()
from=::ffff:10.3.8.30,52426, task_id=33c103f8-5221-448c-8ef3-900af4a0ff1d (api:48)
2024-04-11 11:12:30,611+0200 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health
return={} from=::ffff:10.3.8.30,52426, task_id=33c103f8-5221-448c-8ef3-900af4a0ff1d
(api:54)
2024-04-11 11:12:30,618+0200 INFO (jsonrpc/4) [api.host] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'},
'info': (suppressed)} from=::ffff:10.3.8.30,52426 (api:54)
2024-04-11 11:12:32,056+0200 INFO (migmon/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 94 < than
old value 99, discarded (migration:255)
2024-04-11 11:12:32,057+0200 WARN (migmon/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Migration stalling: remaining
(479MiB) > lowmark (185MiB). (migration:873)
2024-04-11 11:12:32,057+0200 INFO (migmon/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Migration Progress: 470.127 seconds
elapsed, 98% of data processed, total data: 16484MB, processed data: 28862MB, remaining
data: 479MB, transfer speed 496Mbps, zero pages: 515507MB, compressed: 9MB, dirty rate:
13508, memory iteration: 9 (migration:886)
2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,33628,
vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:48)
2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'}
(api:129)
2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [api.virt] FINISH getStats
return={'status': {'code': 1, 'message': "Virtual machine
does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'}"}}
from=::1,33628, vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:54)
2024-04-11 11:12:33,501+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2024-04-11 11:12:33,754+0200 INFO (jsonrpc/2) [vdsm.api] START
repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628,
task_id=16d2af05-faac-4854-89e6-fb3f39452318 (api:48)
2024-04-11 11:12:33,754+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '3.1', 'delay': '0.000296656',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,33628, task_id=16d2af05-faac-4854-89e6-fb3f39452318 (api:54)
2024-04-11 11:12:34,124+0200 INFO (jsonrpc/1) [vdsm.api] START
repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628,
task_id=bdb6aa5e-b9c4-4832-bb8f-fec782cbbde0 (api:48)
2024-04-11 11:12:34,124+0200 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '3.5', 'delay': '0.000296656',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,33628, task_id=bdb6aa5e-b9c4-4832-bb8f-fec782cbbde0 (api:54)
2024-04-11 11:12:34,928+0200 INFO (jsonrpc/4) [vdsm.api] START
getSpmStatus(spUUID='07e52ac2-ab5f-11ec-b531-00163e400079')
from=::ffff:10.3.8.30,52426, task_id=5e808bd5-fefd-454c-a5f6-4785bfad013e (api:48)
2024-04-11 11:12:34,931+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::1,33628
(api:48)
2024-04-11 11:12:34,944+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=())
from=::1,33628, task_id=2e5ceb8f-f518-46eb-88b7-e9caf55de687 (api:48)
2024-04-11 11:12:34,944+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '4.3', 'delay': '0.000296656',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '4.3', 'delay': '0.00024877',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0,
'lastCheck': '0.5', 'delay': '0.000262637',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '4.3', 'delay': '0.000527974',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '4.3', 'delay': '0.000303832',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '4.3', 'delay': '0.00288366',
'valid': True, 'versi
on': 5, 'acquired': True, 'actual': True},
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck':
'4.3', 'delay': '0.000218979', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}} from=::1,33628,
task_id=2e5ceb8f-f518-46eb-88b7-e9caf55de687 (api:54)
2024-04-11 11:12:34,945+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health()
from=::1,33628, task_id=23391530-d739-47ee-8799-d3db76e805ca (api:48)
2024-04-11 11:12:34,945+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health
return={} from=::1,33628, task_id=23391530-d739-47ee-8799-d3db76e805ca (api:54)
2024-04-11 11:12:34,946+0200 INFO (jsonrpc/4) [vdsm.api] FINISH getSpmStatus
return={'spm_st': {'spmStatus': 'SPM', 'spmLver': 13,
'spmId': 2}} from=::ffff:10.3.8.30,52426,
task_id=5e808bd5-fefd-454c-a5f6-4785bfad013e (api:54)
2024-04-11 11:12:34,952+0200 INFO (jsonrpc/6) [api.host] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'},
'info': (suppressed)} from=::1,33628 (api:54)
2024-04-11 11:12:34,989+0200 INFO (jsonrpc/3) [vdsm.api] START
getStoragePoolInfo(spUUID='07e52ac2-ab5f-11ec-b531-00163e400079')
from=::ffff:10.3.8.30,52436, task_id=61162b84-6df1-4916-a959-20214aa7bc2b (api:48)
2024-04-11 11:12:34,992+0200 INFO (jsonrpc/3) [vdsm.api] FINISH getStoragePoolInfo
return={'info': {'domains':
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0:Active,848dac7f-a73d-476f-9371-b54ac8b647de:Active,8ab48aab-a522-4331-ba46-904e13f64d38:Active,c0b21e2c-6b4c-4769-8244-aa69a32c27f2:Active,aa2fa425-8bf3-44b6-b353-81de4e82764a:Active,7f655239-96f9-48f9-99e0-b947220c289e:Active,73e03870-1bad-42d2-bf7b-a55041405c71:Active',
'isoprefix': '', 'lver': 13, 'master_uuid':
'c0b21e2c-6b4c-4769-8244-aa69a32c27f2', 'master_ver': 2, 'name':
'No Description', 'pool_status': 'connected', 'spm_id': 2,
'type': 'FCP', 'version': '5'}, 'dominfo':
{'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'status': 'Active',
'alerts': [], 'isoprefix': '', 'version': 5,
'disktotal': '3298132230144', 'diskfree':
'1233997791232'}, '848dac7f-a73d-476f-9371-b54ac8b647de':
{'status': 'Active', 'alerts': [], 'isoprefix':
'', 'version': 5, 'disktotal': '321719894016',
'diskfree': '316619620352'}, '8ab48aab-a522-4331-ba46-904e13f6
4d38': {'status': 'Active', 'alerts': [],
'isoprefix': '', 'version': 5, 'disktotal':
'2198620602368', 'diskfree': '1906965479424'},
'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'status': 'Active',
'alerts': [], 'isoprefix': '', 'version': 5,
'disktotal': '3298132230144', 'diskfree':
'1311709855744'}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a':
{'status': 'Active', 'alerts': [], 'isoprefix':
'', 'version': 5, 'disktotal': '106971529216',
'diskfree': '63350767616'},
'7f655239-96f9-48f9-99e0-b947220c289e': {'status': 'Active',
'alerts': [], 'isoprefix': '', 'version': 5,
'disktotal': '1099108974592', 'diskfree':
'1071460122624'}, '73e03870-1bad-42d2-bf7b-a55041405c71':
{'status': 'Active', 'alerts': [], 'isoprefix':
'', 'version': 5, 'disktotal': '106971529216',
'diskfree': '11945377792'}}} from=::ffff:10.3.8.30,52436,
task_id=61162b84-6df1-4916-a959-20214aa7bc2b (api:54)
2024-04-11 11:12:35,536+0200 INFO (periodic/2) [vdsm.api] START
getVolumeSize(sdUUID='8ab48aab-a522-4331-ba46-904e13f64d38',
spUUID='07e52ac2-ab5f-11ec-b531-00163e400079',
imgUUID='83f144ac-dfaa-4ede-a0f5-50338f6e1335',
volUUID='da4b0e32-9649-4f7d-834e-e0676a00916b') from=internal,
task_id=d6717cfc-a75c-4f1d-baa9-887711f4732f (api:48)
2024-04-11 11:12:35,536+0200 INFO (periodic/2) [vdsm.api] FINISH getVolumeSize
return={'apparentsize': '110595407872', 'truesize':
'110595407872'} from=internal, task_id=d6717cfc-a75c-4f1d-baa9-887711f4732f
(api:54)
2024-04-11 11:12:35,537+0200 INFO (periodic/1) [vdsm.api] START
getVolumeSize(sdUUID='c0b21e2c-6b4c-4769-8244-aa69a32c27f2',
spUUID='07e52ac2-ab5f-11ec-b531-00163e400079',
imgUUID='f194f2e8-0191-4073-90f6-21e6ddc3e350',
volUUID='f66d26c5-7b48-487a-92d2-dba2d588abff') from=internal,
task_id=f65d5fdc-0b1a-4cc0-9a82-8b0a4c28556b (api:48)
2024-04-11 11:12:35,537+0200 INFO (periodic/1) [vdsm.api] FINISH getVolumeSize
return={'apparentsize': '340376158208', 'truesize':
'340376158208'} from=internal, task_id=f65d5fdc-0b1a-4cc0-9a82-8b0a4c28556b
(api:54)
2024-04-11 11:12:37,549+0200 INFO (jsonrpc/7) [api.host] START getStats() from=::1,33628
(api:48)
2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=())
from=::1,33628, task_id=bd232d01-4552-4937-b2a3-c4b8ec6d9910 (api:48)
2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '6.9', 'delay': '0.000296656',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '2.4', 'delay': '0.000202745',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0,
'lastCheck': '3.1', 'delay': '0.000262637',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '2.4', 'delay': '0.000343811',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '2.4', 'delay': '0.000232965',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '2.4', 'delay': '0.00022068',
'valid': True, 'vers
ion': 5, 'acquired': True, 'actual': True},
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck':
'2.4', 'delay': '0.000188189', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}} from=::1,33628,
task_id=bd232d01-4552-4937-b2a3-c4b8ec6d9910 (api:54)
2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] START multipath_health()
from=::1,33628, task_id=a6903cbf-c31e-45d4-b662-5167cd499faf (api:48)
2024-04-11 11:12:37,562+0200 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health
return={} from=::1,33628, task_id=a6903cbf-c31e-45d4-b662-5167cd499faf (api:54)
2024-04-11 11:12:37,569+0200 INFO (jsonrpc/7) [api.host] FINISH getStats
return={'status': {'code': 0, 'message': 'Done'},
'info': (suppressed)} from=::1,33628 (api:54)
2024-04-11 11:12:40,651+0200 INFO (jsonrpc/0) [api.host] START getAllVmStats()
from=::ffff:10.3.8.30,52426 (api:48)
2024-04-11 11:12:40,651+0200 INFO (jsonrpc/0) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than
old value 99, discarded (migration:255)
2024-04-11 11:12:40,653+0200 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats
return={'status': {'code': 0, 'message': 'Done'},
'statsList': (suppressed)} from=::ffff:10.3.8.30,52426 (api:54)
2024-04-11 11:12:40,867+0200 INFO (periodic/2) [vdsm.api] START repoStats(domains=())
from=internal, task_id=f679c2f0-24e4-454c-b511-653b39c23998 (api:48)
2024-04-11 11:12:40,867+0200 INFO (periodic/2) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '0.2', 'delay': '0.000529618',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '0.2', 'delay': '0.000202745',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0,
'lastCheck': '0.2', 'delay': '0.000262637',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '0.2', 'delay': '0.000343811',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '0.2', 'delay': '0.000232965',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '0.2', 'delay': '0.00022068',
'valid': True, 'ver
sion': 5, 'acquired': True, 'actual': True},
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck':
'0.2', 'delay': '0.000188189', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}} from=internal,
task_id=f679c2f0-24e4-454c-b511-653b39c23998 (api:54)
2024-04-11 11:12:42,059+0200 INFO (migmon/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than
old value 99, discarded (migration:255)
2024-04-11 11:12:42,059+0200 WARN (migmon/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Migration stalling: remaining
(709MiB) > lowmark (185MiB). (migration:873)
2024-04-11 11:12:42,059+0200 WARN (migmon/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Aborting migration (migration:922)
2024-04-11 11:12:42,455+0200 INFO (jsonrpc/1) [api.host] START getAllVmStats()
from=::1,33630 (api:48)
2024-04-11 11:12:42,455+0200 INFO (jsonrpc/1) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than
old value 99, discarded (migration:255)
2024-04-11 11:12:42,456+0200 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats
return={'status': {'code': 0, 'message': 'Done'},
'statsList': (suppressed)} from=::1,33630 (api:54)
2024-04-11 11:12:42,462+0200 INFO (jsonrpc/5) [api.host] START getAllVmIoTunePolicies()
from=::1,33630 (api:48)
2024-04-11 11:12:42,462+0200 INFO (jsonrpc/5) [api.host] FINISH getAllVmIoTunePolicies
return={'status': {'code': 0, 'message': 'Done'},
'io_tune_policies_dict': {'c8e6aafe-1463-4db6-9d3b-76b234f9487d':
{'policy': [], 'current_values': [{'name': 'sdc',
'path':
'/rhev/data-center/mnt/blockSD/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/a57a0a15-1b0a-4f7d-83a0-d59815080caf/d0ba288d-23cf-451f-bd76-d4a7bde6032f',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sda', 'path':
'/rhev/data-center/mnt/blockSD/8ab48aab-a522-4331-ba46-904e13f64d38/images/83f144ac-dfaa-4ede-a0f5-50338f6e1335/da4b0e32-9649-4f7d-834e-e0676a00916b',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}, 'b77300a2-2422-4bfb-a6f6-eea9f4bd4e89':
{'policy': [], 'current_values': [{'name': 'sdc',
'path': '/rhev/data-center/mnt/blockSD
/aa2fa425-8bf3-44b6-b353-81de4e82764a/images/1294d8a3-ad40-4b06-9188-d59f527b921d/f944a7af-cda0-4d75-9c39-7b5063dae0cb',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}, {'name': 'sda', 'path':
'/rhev/data-center/mnt/blockSD/c0b21e2c-6b4c-4769-8244-aa69a32c27f2/images/f194f2e8-0191-4073-90f6-21e6ddc3e350/f66d26c5-7b48-487a-92d2-dba2d588abff',
'ioTune': {'total_bytes_sec': 0, 'read_bytes_sec': 0,
'write_bytes_sec': 0, 'total_iops_sec': 0, 'write_iops_sec': 0,
'read_iops_sec': 0}}]}}} from=::1,33630 (api:54)
2024-04-11 11:12:43,518+0200 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,33628,
vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:48)
2024-04-11 11:12:43,519+0200 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine
does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'}
(api:129)
2024-04-11 11:12:43,519+0200 INFO (jsonrpc/6) [api.virt] FINISH getStats
return={'status': {'code': 1, 'message': "Virtual machine
does not exist: {'vmId': 'cc7931ff-8124-4724-9242-abea2ab5bf42'}"}}
from=::1,33628, vmId=cc7931ff-8124-4724-9242-abea2ab5bf42 (api:54)
2024-04-11 11:12:43,519+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call
VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
2024-04-11 11:12:43,523+0200 ERROR (migsrc/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') operation aborted: migration out
job: canceled by client (migration:331)
2024-04-11 11:12:43,771+0200 INFO (jsonrpc/2) [vdsm.api] START
repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628,
task_id=678887b6-dc37-4ad3-b1e5-b6db22c9f0e4 (api:48)
2024-04-11 11:12:43,771+0200 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '3.1', 'delay': '0.000529618',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,33628, task_id=678887b6-dc37-4ad3-b1e5-b6db22c9f0e4 (api:54)
2024-04-11 11:12:43,839+0200 INFO (migsrc/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Switching from State.STARTED to
State.FAILED (migration:234)
2024-04-11 11:12:43,839+0200 ERROR (migsrc/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Failed to migrate (migration:503)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 477, in
_regular_run
time.time(), machineParams
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 578, in
_startUnderlyingMigration
self._perform_with_conv_schedule(duri, muri)
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 667, in
_perform_with_conv_schedule
self._perform_migration(duri, muri)
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 596, in
_perform_migration
self._migration_flags)
File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 159, in
call
return getattr(self._vm._dom, name)(*a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in
f
ret = attr(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line
131, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2126, in
migrateToURI3
raise libvirtError('virDomainMigrateToURI3() failed')
libvirt.libvirtError: operation aborted: migration out job: canceled by client
2024-04-11 11:12:43,839+0200 INFO (migsrc/c8e6aafe) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') Enabling drive monitoring
(drivemonitor:52)
2024-04-11 11:12:43,890+0200 INFO (jsonrpc/7) [api.virt] START getMigrationStatus()
from=::ffff:10.3.8.30,52426, flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d
(api:48)
2024-04-11 11:12:43,890+0200 INFO (jsonrpc/7) [virt.vm]
(vmId='c8e6aafe-1463-4db6-9d3b-76b234f9487d') new computed progress 98 < than
old value 99, discarded (migration:255)
2024-04-11 11:12:43,890+0200 INFO (jsonrpc/7) [api.virt] FINISH getMigrationStatus
return={'status': {'code': 0, 'message': 'Done'},
'migrationStats': {'status': {'code': 47, 'message':
'Migration canceled'}, 'progress': 99}} from=::ffff:10.3.8.30,52426,
flow_id=681c3996, vmId=c8e6aafe-1463-4db6-9d3b-76b234f9487d (api:54)
2024-04-11 11:12:44,215+0200 INFO (jsonrpc/5) [vdsm.api] START
repoStats(domains=['73e03870-1bad-42d2-bf7b-a55041405c71']) from=::1,33628,
task_id=f98771e0-235b-4c8e-a445-1f72ab61c1d8 (api:48)
2024-04-11 11:12:44,215+0200 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '3.6', 'delay': '0.000529618',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}} from=::1,33628, task_id=f98771e0-235b-4c8e-a445-1f72ab61c1d8 (api:54)
2024-04-11 11:12:44,976+0200 INFO (jsonrpc/6) [api.host] START getStats() from=::1,33628
(api:48)
2024-04-11 11:12:44,989+0200 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=())
from=::1,33628, task_id=f7db90cd-d4ed-4670-83f1-d4336f07cc2e (api:48)
2024-04-11 11:12:44,989+0200 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats
return={'73e03870-1bad-42d2-bf7b-a55041405c71': {'code': 0,
'lastCheck': '4.4', 'delay': '0.000529618',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '8ab48aab-a522-4331-ba46-904e13f64d38': {'code': 0,
'lastCheck': '4.4', 'delay': '0.000202745',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'c0b21e2c-6b4c-4769-8244-aa69a32c27f2': {'code': 0,
'lastCheck': '0.5', 'delay': '0.000303342',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '7f655239-96f9-48f9-99e0-b947220c289e': {'code': 0,
'lastCheck': '4.3', 'delay': '0.000343811',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, '848dac7f-a73d-476f-9371-b54ac8b647de': {'code': 0,
'lastCheck': '4.3', 'delay': '0.000232965',
'valid': True, 'version': 5, 'acquired': True, 'actual':
True}, 'aa2fa425-8bf3-44b6-b353-81de4e82764a': {'code': 0,
'lastCheck': '4.4', 'delay': '0.00022068',
'valid': True, 'vers
ion': 5, 'acquired': True, 'actual': True},
'2a6e7a60-ae0b-4aa2-9b44-1fe9ca8e30b0': {'code': 0, 'lastCheck':
'4.4', 'delay': '0.000188189', 'valid': True,
'version': 5, 'acquired': True, 'actual': True}} from=::1,33628,
task_id=f7db90cd-d4ed-4670-83f1-d4336f07cc2e (api:54)
2024-04-11 11:12:44,990+0200 INFO (jsonrpc/6) [vdsm.api] START multipath_health()
from=::1,33628, task_id=6f2364c5-df82-4104-aed4-e0b71d58ac95 (api:48)
2024-04-11 11:12:44,990+0200 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health
return={} from=::1,33628, task_id=6f2364c5-df82-4104-aed4-e0b71d58ac95 (api:54)