8:03 p.m.
relevant logs taken from Engine server /var/log/ovirt-engine/server.log:
2019-06-28 11:43:26,971-07 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-73) [37ac641] START,
IsVmDuringInitiatingVDSCommand(
IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}),
log id: 4a014d00
2019-06-28 11:43:26,995-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner]
(EE-ManagedThreadFactory-engineScheduled-Thread-73) [37ac641] Exception in
startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of
java.util.LinkedHashMap out of START_ARRAY token
2019-06-28 11:43:28,041-07 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [7a301f69] START,
IsVmDuringInitiatingVDSCommand(
IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}),
log id: 444c67e7
2019-06-28 11:43:28,065-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [7a301f69] Exception in
startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of
java.util.LinkedHashMap out of START_ARRAY token
2019-06-28 11:43:29,122-07 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [4a698430] START,
IsVmDuringInitiatingVDSCommand(
IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}),
log id: 7626e5f1
2019-06-28 11:43:29,146-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [4a698430] Exception in
startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of
java.util.LinkedHashMap out of START_ARRAY token
2019-06-28 11:43:30,198-07 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-90) [fb1b396] START,
IsVmDuringInitiatingVDSCommand(
IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}),
log id: 615cc6a5
2019-06-28 11:43:30,226-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner]
(EE-ManagedThreadFactory-engineScheduled-Thread-90) [fb1b396] Exception in
startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of
java.util.LinkedHashMap out of START_ARRAY token
2019-06-28 11:43:31,284-07 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-5) [4e0f2d55] START,
IsVmDuringInitiatingVDSCommand(
IsVmDuringInitiatingVDSCommandParameters:{vmId='ebe1a8b0-235b-4af0-96a8-e04269cd7a8e'}),
log id: 22ac8448
2019-06-28 11:43:31,312-07 ERROR [org.ovirt.engine.core.bll.HaAutoStartVmsRunner]
(EE-ManagedThreadFactory-engineScheduled-Thread-5) [4e0f2d55] Exception in
startFailedAutoStartVms: JsonMappingException: Can not deserialize instance of
java.util.LinkedHashMap out of START_ARRAY token at [Source:
java.io.StringReader@33492c7c; line: 633, column: 6] (through reference chain:
org.ovirt.engine.core.common.action.VmManagementParametersBase["watchdog"]->org.ovirt.engine.core.common.businessentities.VmWatchdog["specParams"])
Error when attempting to login to administration or vm portal:
org.apache.commons.lang.SerializationException:
org.codehaus.jackson.map.JsonMappingException: Can not deserialize instance of
java.util.LinkedHashMap out of START_ARRAY token at [Source:
java.io.StringReader@11ef98d5; line: 633, column: 6] (through reference chain:
org.ovirt.engine.core.common.action.VmManagementParametersBase["watchdog"]->org.ovirt.engine.core.common.businessentities.VmWatchdog["specParams"])
I've restarted the engine host. I've restarted the engine service. I've
checked for zombie tasks with the db utils - none found.
Engine is hosted in a HA environment with 2 iron boxes, I have rebooted the hosts that
engine is running on. I have migrated hostedengine to both hosts to see if there's an
issue perhaps with one of the hosts. No resolution of issue.
Been searching for various aspects of the logs to try & get an answer (for quite a few
days this issue started some time ago and is in our test environment), all seem to lead to
dead ends.
hosted-engine --vm-status
--== Host 1 (id: 1) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : hostname
Host ID : 1
Engine status : {"reason": "vm not running on this
host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : a3c24020
local_conf_timestamp : 425
Host timestamp : 425
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=425 (Fri Jun 28 14:06:44 2019)
host-id=1
score=3400
vm_conf_refresh_time=425 (Fri Jun 28 14:06:45 2019)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
--== Host 2 (id: 2) status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : hostname
Host ID : 2
Engine status : {"health": "good",
"vm": "up", "detail": "Up"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : e9a6ca22
local_conf_timestamp : 952
Host timestamp : 952
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=952 (Fri Jun 28 14:06:50 2019)
host-id=2
score=3400
vm_conf_refresh_time=952 (Fri Jun 28 14:06:50 2019)
conf_on_shared_storage=True
maintenance=False
state=EngineUp
stopped=False
Last thing I did before this error showed up was created about 10 servers from a template.
There were various lease errors for the servers but they came up and seemed functional.
Clearly the issue is here: "Exception in startFailedAutoStartVms" in the
"START_ARRAY token" but I can't find anything as of yet that says if you see
this - do x,y,z.
Learning oVirt so my apologies if this is a 101 question.