
relevant logs taken from Engine server /var/log/ovirt-engine/server.log: 2019-06-28 11:43:26,971-07 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-73) [37ac641] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}), log id: 4a014d00 2019-06-28 11:43:26,995-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner] (EE-ManagedThreadFactory-engineScheduled-Thread-73) [37ac641] Exception in startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of START_ARRAY token 2019-06-28 11:43:28,041-07 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [7a301f69] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}), log id: 444c67e7 2019-06-28 11:43:28,065-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [7a301f69] Exception in startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of START_ARRAY token 2019-06-28 11:43:29,122-07 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [4a698430] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}), log id: 7626e5f1 2019-06-28 11:43:29,146-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner] (EE-ManagedThreadFactory-engineScheduled-Thread-31) [4a698430] Exception in startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of START_ARRAY token 2019-06-28 11:43:30,198-07 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [fb1b396] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}), log id: 615cc6a5 2019-06-28 11:43:30,226-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner] (EE-ManagedThreadFactory-engineScheduled-Thread-90) [fb1b396] Exception in startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of START_ARRAY token 2019-06-28 11:43:31,284-07 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [4e0f2d55] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}), log id: 22ac8448 2019-06-28 11:43:31,312-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [4e0f2d55] Exception in startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of START_ARRAY token at [Source: java.io.StringReader@33492c7c; line: 633, column: 6] (through reference chain: org.ovirt.engine.core.common.action.VmManagementParametersBase["watchdog"]->org.ovirt.engine.core.common.businessentities.VmWatchdog["specParams"]) Error when attempting to login to administration or vm portal: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of START_ARRAY token at [Source: java.io.StringReader@11ef98d5; line: 633, column: 6] (through reference chain: org.ovirt.engine.core.common.action.VmManagementParametersBase["watchdog"]->org.ovirt.engine.core.common.businessentities.VmWatchdog["specParams"]) I've restarted the engine host. I've restarted the engine service. I've checked for zombie tasks with the db utils - none found. Engine is hosted in a HA environment with 2 iron boxes, I have rebooted the hosts that engine is running on. I have migrated hostedengine to both hosts to see if there's an issue perhaps with one of the hosts. No resolution of issue. Been searching for various aspects of the logs to try & get an answer (for quite a few days this issue started some time ago and is in our test environment), all seem to lead to dead ends. hosted-engine --vm-status --== Host 1 (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : hostname Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : a3c24020 local_conf_timestamp : 425 Host timestamp : 425 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=425 (Fri Jun 28 14:06:44 2019) host-id=1 score=3400 vm_conf_refresh_time=425 (Fri Jun 28 14:06:45 2019) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host 2 (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : hostname Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : e9a6ca22 local_conf_timestamp : 952 Host timestamp : 952 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=952 (Fri Jun 28 14:06:50 2019) host-id=2 score=3400 vm_conf_refresh_time=952 (Fri Jun 28 14:06:50 2019) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False Last thing I did before this error showed up was created about 10 servers from a template. There were various lease errors for the servers but they came up and seemed functional. Clearly the issue is here: "Exception in startFailedAutoStartVms" in the "START_ARRAY token" but I can't find anything as of yet that says if you see this - do x,y,z. Learning oVirt so my apologies if this is a 101 question.