I desperately need assistance/advice.

I am not seeing any specific reason in the logs why I am having this issue.

I tried migrating a VM again to day and basicly got the same results in the log:

Engine Log:

2020-02-24 10:26:16,105Z INFO  [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (default task-62) [c2324230-cb41-4c7f-872a-1437f0a797b1] Running command: MigrateVmToServerCommand internal: false. Entities affected :  ID: 7b0b6e6d-d099-43e0-933f-3c335b54a3a1 Type: VMAction group MIGRATE_VM with role type USER
2020-02-24 10:26:16,171Z INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-62) [c2324230-cb41-4c7f-872a-1437f0a797b1] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='df715653-daf4-457e-839d-95683ab21234', vmId='7b0b6e6d-d099-43e0-933f-3c335b54a3a1', srcHost='host03.timefreight.co.za', dstVdsId='896b7f02-00e9-405c-b166-ec103a7f9ee8', dstHost='host01.timefreight.co.za:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='62', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='172.10.10.1'}), log id: 185fa2b8
2020-02-24 10:26:16,172Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-62) [c2324230-cb41-4c7f-872a-1437f0a797b1] START, MigrateBrokerVDSCommand(HostName = host03.timefreight.co.za, MigrateVDSCommandParameters:{hostId='df715653-daf4-457e-839d-95683ab21234', vmId='7b0b6e6d-d099-43e0-933f-3c335b54a3a1', srcHost='host03.timefreight.co.za', dstVdsId='896b7f02-00e9-405c-b166-ec103a7f9ee8', dstHost='host01.timefreight.co.za:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='62', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='172.10.10.1'}), log id: 6157c8c9
2020-02-24 10:26:16,179Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-62) [c2324230-cb41-4c7f-872a-1437f0a797b1] FINISH, MigrateBrokerVDSCommand, return: , log id: 6157c8c9
2020-02-24 10:26:16,182Z INFO  [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-62) [c2324230-cb41-4c7f-872a-1437f0a797b1] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 185fa2b8
2020-02-24 10:26:16,190Z INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-62) [c2324230-cb41-4c7f-872a-1437f0a797b1] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: accpac, Source: host03.timefreight.co.za, Destination: host01.timefreight.co.za, User: admin@internal-authz).
2020-02-24 10:26:16,195Z INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] VM '7b0b6e6d-d099-43e0-933f-3c335b54a3a1'(accpac) moved from 'MigratingFrom' --> 'Up'
2020-02-24 10:26:16,195Z INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-13) [] Adding VM '7b0b6e6d-d099-43e0-933f-3c335b54a3a1'(accpac) to re-run list
2020-02-24 10:26:16,198Z ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-13) [] Rerun VM '7b0b6e6d-d099-43e0-933f-3c335b54a3a1'. Called from VDS 'host03.timefreight.co.za'
2020-02-24 10:26:16,201Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-207443) [] START, MigrateStatusVDSCommand(HostName = host03.timefreight.co.za, MigrateStatusVDSCommandParameters:{hostId='df715653-daf4-457e-839d-95683ab21234', vmId='7b0b6e6d-d099-43e0-933f-3c335b54a3a1'}), log id: 3b6274fe
2020-02-24 10:26:16,205Z INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-207443) [] FINISH, MigrateStatusVDSCommand, return: , log id: 3b6274fe
2020-02-24 10:26:16,226Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-207443) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed  (VM: accpac, Source: host03.timefreight.co.za, Destination: host01.timefreight.co.za).
2020-02-24 10:26:16,230Z INFO  [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (EE-ManagedThreadFactory-engine-Thread-207443) [] Lock freed to object 'EngineLock:{exclusiveLocks='[7b0b6e6d-d099-43e0-933f-3c335b54a3a1=VM]', sharedLocks=''}'

New host VDSM Log:

2020-02-24 10:26:14,201+0000 INFO  (jsonrpc/1) [api.host] START getStats() from=::ffff:172.10.10.50,47246 (api:48)
2020-02-24 10:26:14,219+0000 INFO  (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::ffff:172.10.10.50,47246, task_id=b2473290-b0fc-4af9-afbc-792ab871f326 (api:48)
2020-02-24 10:26:14,219+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH repoStats return={u'aea451d8-a347-40d4-be20-3bb8f612e7cd': {'code': 0, 'actual': True, 'version': 5, 'acquired': True, 'delay': '0.000477228', 'lastCheck': '7.2', 'valid': True}, u'e13a1137-3bc1-4d12-a623-60941e6c29b1': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000578509', 'lastCheck': '7.1', 'valid': True}, u'8969a4be-de76-4fd0-93d5-ba6dff95c9c1': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000363948', 'lastCheck': '8.8', 'valid': True}, u'988ccfea-6cca-4c78-8a0d-9b4687b05a6e': {'code': 0, 'actual': True, 'version': 5, 'acquired': True, 'delay': '0.000423126', 'lastCheck': '6.7', 'valid': True}} from=::ffff:172.10.10.50,47246, task_id=b2473290-b0fc-4af9-afbc-792ab871f326 (api:54)
2020-02-24 10:26:14,220+0000 INFO  (jsonrpc/1) [vdsm.api] START multipath_health() from=::ffff:172.10.10.50,47246, task_id=567018ae-b772-4eb1-a170-6b0ad0cef513 (api:48)
2020-02-24 10:26:14,220+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::ffff:172.10.10.50,47246, task_id=567018ae-b772-4eb1-a170-6b0ad0cef513 (api:54)
2020-02-24 10:26:14,225+0000 INFO  (jsonrpc/1) [api.host] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'info': {'cpuStatistics': {'20': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '21': {'cpuUser': '0.20', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '99.80'}, '22': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '23': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '1': {'cpuUser': '0.40', 'nodeIndex': 1, 'cpuSys': '0.60', 'cpuIdle': '99.00'}, '0': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.86'}, '3': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '2': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}, '5': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '4': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '7': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '6': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '9': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '8': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '11': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '10': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '13': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '12': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '15': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '14': {'cpuUser': '0.27', 'nodeIndex': 0, 'cpuSys': '0.13', 'cpuIdle': '99.60'}, '17': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '16': {'cpuUser': '0.00', 'nodeIndex': 0, 'cpuSys': '0.07', 'cpuIdle': '99.93'}, '19': {'cpuUser': '0.00', 'nodeIndex': 1, 'cpuSys': '0.00', 'cpuIdle': '100.00'}, '18': {'cpuUser': '0.07', 'nodeIndex': 0, 'cpuSys': '0.00', 'cpuIdle': '99.93'}}, 'numaNodeMemFree': {'1': {'memPercent': 4, 'memFree': '47261'}, '0': {'memPercent': 5, 'memFree': '46860'}}, 'memShared': 0, 'thpState': 'always', 'ksmMergeAcrossNodes': True, 'vmCount': 0, 'memUsed': '2', 'storageDomains': {u'aea451d8-a347-40d4-be20-3bb8f612e7cd': {'code': 0, 'actual': True, 'version': 5, 'acquired': True, 'delay': '0.000477228', 'lastCheck': '7.2', 'valid': True}, u'8969a4be-de76-4fd0-93d5-ba6dff95c9c1': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000363948', 'lastCheck': '8.8', 'valid': True}, u'e13a1137-3bc1-4d12-a623-60941e6c29b1': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000578509', 'lastCheck': '7.1', 'valid': True}, u'988ccfea-6cca-4c78-8a0d-9b4687b05a6e': {'code': 0, 'actual': True, 'version': 5, 'acquired': True, 'delay': '0.000423126', 'lastCheck': '6.7', 'valid': True}}, 'incomingVmMigrations': 0, 'network': {'p4p1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'p4p1', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '0', 'state': 'down'}, 'p4p2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'p4p2', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '0', 'state': 'down'}, 'bond0': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'bond0', 'tx': '2686605379', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1582539974.218536, 'rx': '54169240727', 'state': 'up'}, 'ovirtmgmt': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '845', 'name': 'ovirtmgmt', 'tx': '2575785594', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '52484350965', 'state': 'up'}, 'lo': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'lo', 'tx': '73036971', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '73036971', 'state': 'up'}, 'ovs-system': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'ovs-system', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '0', 'state': 'down'}, ';vdsmdummy;': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': ';vdsmdummy;', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '0', 'state': 'down'}, 'br-int': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'br-int', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '0', 'state': 'down'}, 'em1': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'em1', 'tx': '2696308698', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1582539974.218536, 'rx': '54026312941', 'state': 'up'}, 'genev_sys_6081': {'rxErrors': '0', 'txErrors': '11', 'speed': '1000', 'rxDropped': '0', 'name': 'genev_sys_6081', 'tx': '0', 'txDropped': '0', 'duplex': 'unknown', 'sampleTime': 1582539974.218536, 'rx': '0', 'state': 'up'}, 'em2': {'rxErrors': '0', 'txErrors': '0', 'speed': '1000', 'rxDropped': '0', 'name': 'em2', 'tx': '384', 'txDropped': '0', 'duplex': 'full', 'sampleTime': 1582539974.218536, 'rx': '331164676', 'state': 'up'}}, 'txDropped': '845', 'anonHugePages': '212', 'ksmPages': 100, 'elapsedTime': '434870.25', 'cpuLoad': '0.01', 'cpuSys': '0.04', 'diskStats': {'/var/log': {'free': '7434'}, '/var/run/vdsm/': {'free': '48186'}, '/tmp': {'free': '905'}}, 'cpuUserVdsmd': '0.47', 'netConfigDirty': 'False', 'memCommitted': 0, 'ksmState': False, 'vmMigrating': 0, 'ksmCpu': 0, 'memAvailable': 95451, 'bootTime': '1582103448', 'haStats': {'active': False, 'configured': False, 'score': 0, 'localMaintenance': False, 'globalMaintenance': False}, 'momStatus': 'active', 'multipathHealth': {}, 'rxDropped': '0', 'outgoingVmMigrations': 0, 'swapTotal': 4095, 'swapFree': 4095, 'hugepages': defaultdict(<type 'dict'>, {1048576: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}, 2048: {'resv_hugepages': 0, 'free_hugepages': 0, 'nr_overcommit_hugepages': 0, 'surplus_hugepages': 0, 'vm.free_hugepages': 0, 'nr_hugepages': 0, 'nr_hugepages_mempolicy': 0}}), 'dateTime': '2020-02-24T10:26:14 GMT', 'cpuUser': '0.05', 'memFree': 95195, 'cpuIdle': '99.91', 'vmActive': 0, 'v2vJobs': {}, 'cpuSysVdsmd': '0.33'}} from=::ffff:172.10.10.50,47246 (api:54)
2020-02-24 10:26:14,228+0000 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats succeeded in 0.03 seconds (__init__:312)