Hi all,
A few weeks ago I did a clean install of the latest oVirt-4.3.2 and
imported some VMs from oVirt-3. Three nodes running oVirt Node and oVirt
Engine installed on a separate system.
I noticed that some times some VMs will boot successfully but the Web UI
will still show "Powering UP" for days after the VM has been up. I can
power down the VM and power back up and it may update the Web UI status to
UP.
While debugging the above issue I noticed that some VMs will trigger errors
during boot. I can power on a VM on one node, see the errors below started
happening every 4-5 seconds, then power down the VM, errors stop, then
power up the VM on a different node without a problem. Another VM though
may trigger the errors on the same node.
Everything is very inconsistent. I can't find a pattern. I tried different
VMs, different nodes, and I'm getting mixed results. Hopefully the errors
will give some clue.
Here is what I'm seeing scrolling every 4-5 seconds:
-------------------------
On oVirt Node:
==> vdsm.log <==
2019-04-12 10:50:31,543-0700 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer]
Internal server error (__init__:350)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345,
in _handle_request
res = method(**params)
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in
_dynamicMethod
result = fn(*methodArgs)
File "<string>", line 2, in getAllVmStats
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in
getAllVmStats
statsList = self._cif.getAllVmStats()
File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in
getAllVmStats
return [v.getStats() for v in self.vmContainer.values()]
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in
getStats
oga_stats = self._getGuestStats()
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in
_getGuestStats
stats = self.guestAgent.getGuestInfo()
File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line
505, in getGuestInfo
del qga['appsList']
KeyError: 'appsList'
==> mom.log <==
2019-04-12 10:50:31,547 - mom.VdsmRpcBase - ERROR - Command
Host.getAllVmStats with args {} failed:
(code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"})
----------------------
On oVirt Engine
2019-04-12 10:50:35,692-07 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"'appsList'"}]
2019-04-12 10:50:35,693-07 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Failed in
'GetAllVmStatsVDS' method
2019-04-12 10:50:35,693-07 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Command
'GetAllVmStatsVDSCommand(HostName = sdod-ovnode-03,
VdsIdVDSCommandParametersBase:{hostId='12e38ad3-6327-4c94-8be4-88912d283729'})'
execution failed: VDSGenericException: VDSErrorException: Failed to
GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason':
"'appsList'"}, code = -32603
Thank you,
-- Peter
