On Tue, Jun 19, 2018 at 11:39 AM <rni@chef.net> wrote:

Hello,

I'm using a 4 node cluster oVirt 4.1. After updating the Engine to 4.2.3.8 (and ContOS 7.5), I tried to update the nodes too.
Unfortunally, the live migration is not working (any more). Maybe it's related to following error found in engine.log

2018-06-19 09:32:53,469+02 WARN [org.ovirt.engine.core.bll.provider.network.openstack.CustomizedRESTEasyConnector] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [4d6a7dc4] Cannot register external providers trust store: java.io.IOException: Keystore was tampered with, or password was incorrect
2018-06-19 09:32:53,475+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [4d6a7dc4] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error unable to find valid certification path to requested target and code 5050)

Didi, do have an idea what can be the issue?

I'm using the selfsigned CA provided with oVirt.

Nodes (source and destination)
OS-Version:RHEL - 7 - 4.1708.el7.centos
Kernelversion: 3.10.0 - 693.21.1.el7.x86_64
KVM-Version:2.9.0 - 16.el7_4.14.1
LIBVIRT-Version:libvirt-3.2.0-14.el7_4.9
VDSM-Version:vdsm-4.19.45-1.el7.centos
SPICE-Version:0.12.8 - 2.el7.1
CEPH-Version:librbd1-0.94.5-2.el7
Kernel Features:PTI: 1, IBPB: 0, IBRS: 0

engine.log for the mirgration:
2018-06-14 13:13:30,300+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-246968) [] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Migration initiated by system (VM: vm_to_migrate, Source: SOURCE, Destination: DESTINATION, Reason: Host preparing for maintenance).
2018-06-14 13:13:32,886+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] Fetched 8 VMs from VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'
2018-06-14 13:13:32,887+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:13:32,887+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:13:47,939+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-42) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:13:47,939+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-42) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:14:02,993+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:14:02,993+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:14:18,042+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:14:18,042+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:14:33,090+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:14:33,090+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:14:38,356+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d1457e7e-f29d-49ac-b635-2457a7cbc59f=PROVIDER]', sharedLocks=''}'
2018-06-14 13:14:38,369+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Running command: SyncNetworkProviderCommand internal: true.
2018-06-14 13:14:38,370+02 WARN [org.ovirt.engine.core.bll.provider.network.openstack.CustomizedRESTEasyConnector] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Cannot register external providers trust store: java.io.IOException: Keystore was tampered with, or password was incorrect
2018-06-14 13:14:38,379+02 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error unable to find valid certification path to requested target and code 5050)
2018-06-14 13:14:38,382+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [6d991f10] Lock freed to object 'EngineLock:{exclusiveLocks='[d1457e7e-f29d-49ac-b635-2457a7cbc59f=PROVIDER]', sharedLocks=''}'
2018-06-14 13:14:48,139+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:14:48,139+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:15:03,193+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:15:03,193+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-71) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:15:18,242+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:15:18,242+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-63) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:15:33,293+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) was unexpectedly detected as 'MigratingTo' on VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) (expected on 'ea7bae8f-d966-4baa-b8b4-a5522b88ec3a')
2018-06-14 13:15:33,293+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-17) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111' is migrating to VDS 'f3749f1e-a68d-424f-aa3a-f0ec996df498'(DESTINATION) ignoring it in the refresh until migration is done
2018-06-14 13:15:45,377+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [] VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) moved from 'MigratingFrom' --> 'Up'
2018-06-14 13:15:45,377+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [] Adding VM '4aff4193-ba75-481d-92b3-59b62cd8b111'(vm_to_migrate) to re-run list
2018-06-14 13:15:45,400+02 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-10) [] Rerun VM '4aff4193-ba75-481d-92b3-59b62cd8b111'. Called from VDS 'SOURCE'
2018-06-14 13:15:45,451+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] START, MigrateStatusVDSCommand(HostName = SOURCE, MigrateStatusVDSCommandParameters:{hostId='ea7bae8f-d966-4baa-b8b4-a5522b88ec3a', vmId='4aff4193-ba75-481d-92b3-59b62cd8b111'}), log id: 25f95334
2018-06-14 13:15:45,453+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] FINISH, MigrateStatusVDSCommand, log id: 25f95334
2018-06-14 13:15:45,478+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-247023) [] EVENT_ID: VM_MIGRATION_TRYING_RERUN(128), Failed to migrate VM vm_to_migrate to Host DESTINATION . Trying to migrate to another Host.
2018-06-14 13:15:45,557+02 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] Running command: MigrateVmCommand internal: true. Entities affected : ID: 4aff4193-ba75-481d-92b3-59b62cd8b111 Type: VMAction group MIGRATE_VM with role type USER
2018-06-14 13:15:45,595+02 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='ea7bae8f-d966-4baa-b8b4-a5522b88ec3a', vmId='4aff4193-ba75-481d-92b3-59b62cd8b111', srcHost='10.157.8.37', dstVdsId='69ebacd7-ee55-4c52-abf1-437a14d5fb0d', dstHost='10.157.8.42:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='false', migrateCompressed='false', consoleAddress='null', maxBandwidth='500', enableGuestEvents='false', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='null', dstQemu='192.168.1.113'}), log id: 736d023f
2018-06-14 13:15:45,595+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] START, MigrateBrokerVDSCommand(HostName = SOURCE, MigrateVDSCommandParameters:{hostId='ea7bae8f-d966-4baa-b8b4-a5522b88ec3a', vmId='4aff4193-ba75-481d-92b3-59b62cd8b111', srcHost='10.157.8.37', dstVdsId='69ebacd7-ee55-4c52-abf1-437a14d5fb0d', dstHost='10.157.8.42:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='false', migrateCompressed='false', consoleAddress='null', maxBandwidth='500', enableGuestEvents='false', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='null', dstQemu='192.168.1.113'}), log id: 58572c5b
2018-06-14 13:15:45,601+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] FINISH, MigrateBrokerVDSCommand, log id: 58572c5b
2018-06-14 13:15:45,604+02 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (EE-ManagedThreadFactory-engine-Thread-247023) [] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 736d023f

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/B6IEVX5XEBFHJCWCHGRMPOJ27VUJI2TL/