Hi all,

A few weeks ago I did a clean install of the latest oVirt-4.3.2 and imported some VMs from oVirt-3. Three nodes running oVirt Node and oVirt Engine installed on a separate system.

I noticed that some times some VMs will boot successfully but the Web UI will still show "Powering UP" for days after the VM has been up. I can power down the VM and power back up and it may update the Web UI status to UP.

While debugging the above issue I noticed that some VMs will trigger errors during boot. I can power on a VM on one node, see the errors below started happening every 4-5 seconds, then power down the VM, errors stop, then power up the VM on a different node without a problem. Another VM though may trigger the errors on the same node.

Everything is very inconsistent. I can't find a pattern. I tried different VMs, different nodes, and I'm getting mixed results. Hopefully the errors will give some clue.

Here is what I'm seeing scrolling every 4-5 seconds:

-------------------------

On oVirt Node:

==> vdsm.log <==
2019-04-12 10:50:31,543-0700 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod
    result = fn(*methodArgs)
  File "<string>", line 2, in getAllVmStats
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats
    statsList = self._cif.getAllVmStats()
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats
    return [v.getStats() for v in self.vmContainer.values()]
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats
    oga_stats = self._getGuestStats()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats
    stats = self.guestAgent.getGuestInfo()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo
    del qga['appsList']
KeyError: 'appsList'

==> mom.log <==
2019-04-12 10:50:31,547 - mom.VdsmRpcBase - ERROR - Command Host.getAllVmStats with args {} failed:
(code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"})

----------------------

On oVirt Engine

2019-04-12 10:50:35,692-07 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}]
2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Failed in 'GetAllVmStatsVDS' method
2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Command 'GetAllVmStatsVDSCommand(HostName = sdod-ovnode-03, VdsIdVDSCommandParametersBase:{hostId='12e38ad3-6327-4c94-8be4-88912d283729'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'appsList'"}, code = -32603

Thank you,

-- Peter