After upgrade to vdsm-4.40.90.4-1.el8 - Internal JSON-RPC error - how to fix?
Hi, After we upgraded to vdsm-4.40.90.4-1.el8 on our two node cluster two things has happned: 1. First node that was upgraded now continuously logs this error: VDSM <node name> command Get Host Statistics failed: Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"} 2. During import of two virtual machines from VMware (has done multible before upgrade) the import seems to never finish - currently running on day 2-3... any clues how to fix this? /John
I also got the error when I try to import an ova from vmware to my ovirt cluster using a san storage domain. I resovled this by importing this ova to a standalone host which is using its local storage. 在 2021/11/18 17:41, John Mortensen 写道:
On Thu, Jan 6, 2022 at 9:20 AM Adam Xu <adam_xu@adagene.com.cn> wrote: those errors. Can you please share the import logs? [1] https://bugzilla.redhat.com/show_bug.cgi?id=2026809
在 2022/1/6 15:53, Liran Rotenberg 写道:
when I try to import ova to san storage. there are tons of log like: 2022-01-06 09:15:55,058+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt4.adagene.cn command Get Host Statistics failed: Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"} 2022-01-06 09:15:55,059+08 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-8) [] Unable to GetStats: VDSErrorException: VDSGenericException: VDSErrorException: Failed to Get Host Statistics, error = Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}, code = -32603 2022-01-06 09:15:58,347+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-93) [b8ca5e99-e3e5-4a09-8b66-e225f45fdf77] Command 'ImportVmFromOva' (id: 'a1ef50bc-5dba-40af-9ebd-48bb421db495') waiting on child command id: '7943bb90-5964-4458-a1b8-d957cff94f09' type:'ConvertOva' to complete 2022-01-06 09:16:08,358+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-88) [b8ca5e99-e3e5-4a09-8b66-e225f45fdf77] Command 'ImportVmFromOva' (id: 'a1ef50bc-5dba-40af-9ebd-48bb421db495') waiting on child command id: '7943bb90-5964-4458-a1b8-d957cff94f09' type:'ConvertOva' to complete 2022-01-06 09:16:10,090+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 09:16:10,090+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [] Failed in 'Get Host Statistics' method 2022-01-06 09:16:10,090+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 09:16:10,092+08 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-13) [] Unable to GetStats: VDSErrorException: VDSGenericException: VDSErrorException: Failed to Get Host Statistics, error = Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}, code = -32603 2022-01-06 09:16:18,367+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-30) [b8ca5e99-e3e5-4a09-8b66-e225f45fdf77] Command 'ImportVmFromOva' (id: 'a1ef50bc-5dba-40af-9ebd-48bb421db495') waiting on child command id: '7943bb90-5964-4458-a1b8-d957cff94f09' type:'ConvertOva' to complete 2022-01-06 09:16:25,099+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-34) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 09:16:25,099+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-34) [] Failed in 'Get Host Statistics' method 2022-01-06 09:16:25,099+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-34) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] these logs seems endless, I have to stop importing tasks after several hours waitting. --------------------------------------------------------------------------- when I try to import ova to a standalone host with local storage, logs like: 2022-01-06 12:31:21,737+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadP ool-Thread-9) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ovirt6.adagene.cn command Get Host Statistics failed: Internal JSON-RPC error: {'reason': "'st r' object has no attribute 'decode'"} 2022-01-06 12:31:21,744+08 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread- 9) [] Unable to GetStats: VDSErrorException: VDSGenericException: VDSErrorException: Failed to Get Host Statistics, error = Internal JSON-RPC error: {'reason': "' str' object has no attribute 'decode'"}, code = -32603 2022-01-06 12:31:23,776+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Th read-47) [9015b315-7fd1-408b-a231-41354d4b6384] Command 'ImportVmFromOva' (id: 'c8b98e81-bfc7-4e51-8082-4e4aec0d182e') waiting on child command id: '1378cdc5-a8d4 -4588-94d2-a72004cf268d' type:'ConvertOva' to complete 2022-01-06 12:31:33,788+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Th read-77) [9015b315-7fd1-408b-a231-41354d4b6384] Command 'ImportVmFromOva' (id: 'c8b98e81-bfc7-4e51-8082-4e4aec0d182e') waiting on child command id: '1378cdc5-a8d4 -4588-94d2-a72004cf268d' type:'ConvertOva' to complete 2022-01-06 12:31:36,709+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-85) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 12:31:36,710+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-85) [] Failed in 'Get Host Statistics' method 2022-01-06 12:31:36,710+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-85) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 12:31:36,712+08 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread- 85) [] Unable to GetStats: VDSErrorException: VDSGenericException: VDSErrorException: Failed to Get Host Statistics, error = Internal JSON-RPC error: {'reason': " 'str' object has no attribute 'decode'"}, code = -32603 2022-01-06 12:31:43,800+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Th read-86) [9015b315-7fd1-408b-a231-41354d4b6384] Command 'ImportVmFromOva' (id: 'c8b98e81-bfc7-4e51-8082-4e4aec0d182e') waiting on child command id: '1378cdc5-a8d4 -4588-94d2-a72004cf268d' type:'ConvertOva' to complete 2022-01-06 12:31:51,712+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-36) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 12:31:51,712+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-36) [] Failed in 'Get Host Statistics' method 2022-01-06 12:31:51,713+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-36) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 12:31:51,714+08 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread- 36) [] Unable to GetStats: VDSErrorException: VDSGenericException: VDSErrorException: Failed to Get Host Statistics, error = Internal JSON-RPC error: {'reason': " 'str' object has no attribute 'decode'"}, code = -32603 2022-01-06 12:31:53,812+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Th read-93) [9015b315-7fd1-408b-a231-41354d4b6384] Command 'ImportVmFromOva' (id: 'c8b98e81-bfc7-4e51-8082-4e4aec0d182e') waiting on child command id: '1378cdc5-a8d4 -4588-94d2-a72004cf268d' type:'ConvertOva' to complete 2022-01-06 12:32:03,823+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Th read-4) [9015b315-7fd1-408b-a231-41354d4b6384] Command 'ImportVmFromOva' (id: 'c8b98e81-bfc7-4e51-8082-4e4aec0d182e') waiting on child command id: '1378cdc5-a8d4- 4588-94d2-a72004cf268d' type:'ConvertOva' to complete 2022-01-06 12:32:06,816+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-76) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] 2022-01-06 12:32:06,816+08 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-76) [] Failed in 'Get Host Statistics' method 2022-01-06 12:32:06,817+08 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool -Thread-76) [] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"}] And it will succeed after serveral warning messages above.
On Fri, Jan 7, 2022 at 2:52 AM Adam Xu <adam_xu@adagene.com.cn> wrote:
These are spam logs from the bug, nothing that harms the import. As you can see: 2022-01-06 09:16:18,367+08 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-30) [b8ca5e99-e3e5-4a09-8b66-e225f45fdf77] Command 'ImportVmFromOva' (id: 'a1ef50bc-5dba-40af-9ebd-48bb421db495') waiting on child command id: '7943bb90-5964-4458-a1b8-d957cff94f09' type:'ConvertOva' to complete The import is still running, now it converts the OVA. Import logs might show us more, can be found in the host importing, under /var/log/vdsm/import.
participants (3)
-
Adam Xu -
John Mortensen -
Liran Rotenberg