oVirt 4.3.2.1-1.el7 Errors at VM boot
Hi all, A few weeks ago I did a clean install of the latest oVirt-4.3.2 and imported some VMs from oVirt-3. Three nodes running oVirt Node and oVirt Engine installed on a separate system. I noticed that some times some VMs will boot successfully but the Web UI will still show "Powering UP" for days after the VM has been up. I can power down the VM and power back up and it may update the Web UI status to UP. While debugging the above issue I noticed that some VMs will trigger errors during boot. I can power on a VM on one node, see the errors below started happening every 4-5 seconds, then power down the VM, errors stop, then power up the VM on a different node without a problem. Another VM though may trigger the errors on the same node. Everything is very inconsistent. I can't find a pattern. I tried different VMs, different nodes, and I'm getting mixed results. Hopefully the errors will give some clue. Here is what I'm seeing scrolling every 4-5 seconds: ------------------------- On oVirt Node: ==> vdsm.log <== 2019-04-12 10:50:31,543-0700 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList' ==> mom.log <== 2019-04-12 10:50:31,547 - mom.VdsmRpcBase - ERROR - Command Host.getAllVmStats with args {} failed: (code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}) ---------------------- On oVirt Engine 2019-04-12 10:50:35,692-07 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}] 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Failed in 'GetAllVmStatsVDS' method 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Command 'GetAllVmStatsVDSCommand(HostName = sdod-ovnode-03, VdsIdVDSCommandParametersBase:{hostId='12e38ad3-6327-4c94-8be4-88912d283729'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'appsList'"}, code = -32603 Thank you, -- Peter
On Fri, Apr 12, 2019 at 7:57 PM Wood Peter <peterwood.sd@gmail.com> wrote:
Hi all,
A few weeks ago I did a clean install of the latest oVirt-4.3.2 and imported some VMs from oVirt-3. Three nodes running oVirt Node and oVirt Engine installed on a separate system.
I noticed that some times some VMs will boot successfully but the Web UI will still show "Powering UP" for days after the VM has been up. I can power down the VM and power back up and it may update the Web UI status to UP.
While debugging the above issue I noticed that some VMs will trigger errors during boot. I can power on a VM on one node, see the errors below started happening every 4-5 seconds, then power down the VM, errors stop, then power up the VM on a different node without a problem. Another VM though may trigger the errors on the same node.
Everything is very inconsistent. I can't find a pattern. I tried different VMs, different nodes, and I'm getting mixed results. Hopefully the errors will give some clue.
Here is what I'm seeing scrolling every 4-5 seconds:
------------------------- On oVirt Node:
==> vdsm.log <== 2019-04-12 10:50:31,543-0700 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList'
==> mom.log <== 2019-04-12 10:50:31,547 - mom.VdsmRpcBase - ERROR - Command Host.getAllVmStats with args {} failed: (code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"})
This was tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1690301 and it should be fixed with vdsm-4.30.12; can you please report which VDSM version are you using on your host?
---------------------- On oVirt Engine
2019-04-12 10:50:35,692-07 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}] 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Failed in 'GetAllVmStatsVDS' method 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Command 'GetAllVmStatsVDSCommand(HostName = sdod-ovnode-03, VdsIdVDSCommandParametersBase:{hostId='12e38ad3-6327-4c94-8be4-88912d283729'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'appsList'"}, code = -32603
Thank you, -- Peter _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EE22ADRNIFF6UP...
Thank you. I'm using version 4.30.11 On Mon, Apr 15, 2019 at 12:12 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Fri, Apr 12, 2019 at 7:57 PM Wood Peter <peterwood.sd@gmail.com> wrote:
Hi all,
A few weeks ago I did a clean install of the latest oVirt-4.3.2 and imported some VMs from oVirt-3. Three nodes running oVirt Node and oVirt Engine installed on a separate system.
I noticed that some times some VMs will boot successfully but the Web UI will still show "Powering UP" for days after the VM has been up. I can power down the VM and power back up and it may update the Web UI status to UP.
While debugging the above issue I noticed that some VMs will trigger errors during boot. I can power on a VM on one node, see the errors below started happening every 4-5 seconds, then power down the VM, errors stop, then power up the VM on a different node without a problem. Another VM though may trigger the errors on the same node.
Everything is very inconsistent. I can't find a pattern. I tried different VMs, different nodes, and I'm getting mixed results. Hopefully the errors will give some clue.
Here is what I'm seeing scrolling every 4-5 seconds:
------------------------- On oVirt Node:
==> vdsm.log <== 2019-04-12 10:50:31,543-0700 ERROR (jsonrpc/3) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getAllVmStats File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats oga_stats = self._getGuestStats() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats stats = self.guestAgent.getGuestInfo() File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo del qga['appsList'] KeyError: 'appsList'
==> mom.log <== 2019-04-12 10:50:31,547 - mom.VdsmRpcBase - ERROR - Command Host.getAllVmStats with args {} failed: (code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"})
This was tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1690301 and it should be fixed with vdsm-4.30.12; can you please report which VDSM version are you using on your host?
---------------------- On oVirt Engine
2019-04-12 10:50:35,692-07 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'appsList'"}] 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Failed in 'GetAllVmStatsVDS' method 2019-04-12 10:50:35,693-07 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-53) [] Command 'GetAllVmStatsVDSCommand(HostName = sdod-ovnode-03, VdsIdVDSCommandParametersBase:{hostId='12e38ad3-6327-4c94-8be4-88912d283729'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Internal JSON-RPC error: {'reason': "'appsList'"}, code = -32603
Thank you, -- Peter _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EE22ADRNIFF6UP...
participants (2)
-
Simone Tiraboschi -
Wood Peter