Problem solved.
I used Ansible and ran 'grep Duplicate /var/log/vdsm/vdsm.log' on all 121 of my
hosts. Only the log of the host currently running the hosted-engine had entries.
I migrated the hosted-engine, and all other VMs to other hosts, and then put the affected
host into maintenance. I then undeployed it; then patched it; and finally deployed it.
Its back up and running and taking VMs, and no more 'Duplicate keys' errors.
-----Original Message-----
From: Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com>
Sent: Monday, April 26, 2021 7:18 PM
To: Nir Soffer <nsoffer(a)redhat.com>
Cc: users(a)ovirt.org
Subject: [ovirt-users] Re: Attempting to detach a storage domain
There is no /var/log/vdsm directory on the hosted-engine.
This is from the current host, hosting the engine: aaa.bbb.ccc.100 is the hosted-engine
2021-04-26 19:12:42,760-0500 INFO (jsonrpc/2) [api.host] START getCapabilities()
from=::ffff:aaa.bbb.ccc.100,37812 (api:48)
2021-04-26 19:12:42,763-0500 INFO (jsonrpc/2) [api.host] FINISH getCapabilities
error=internal error: Duplicate key from=::ffff:168.127.151.100,37812 (api:52)
2021-04-26 19:12:42,763-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server
error (__init__:350) Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request
res = method(**params)
File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod
result = fn(*methodArgs)
File "<string>", line 2, in getCapabilities
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1371, in
getCapabilities
c = caps.get()
File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 95, in get
machinetype.compatible_cpu_models())
File "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 43, in
__call__
value = self.func(*args)
File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 142, in
compatible_cpu_models
all_models = domain_cpu_models(c, arch, cpu_mode)
File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 97, in
domain_cpu_models
domcaps = conn.getDomainCapabilities(None, arch, None, virt_type, 0)
File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line
131, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3844, in
getDomainCapabilities
if ret is None: raise libvirtError ('virConnectGetDomainCapabilities()
failed', conn=self)
libvirtError: internal error: Duplicate key
2021-04-26 19:12:42,764-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call
Host.getCapabilities failed (error -32603) in 0.00 seconds (__init__:312)
-----Original Message-----
From: Nir Soffer <nsoffer(a)redhat.com>
Sent: Monday, April 26, 2021 5:12 PM
To: Stier, Matthew <Matthew.Stier(a)fujitsu.com>
Cc: Benny Zlotnik <bzlotnik(a)redhat.com>; users(a)ovirt.org
Subject: Re: [ovirt-users] Re: Attempting to detach a storage domain
On Tue, Apr 27, 2021 at 12:48 AM Matthew.Stier(a)fujitsu.com
<Matthew.Stier(a)fujitsu.com> wrote:
I'm getting tons of these in the
hosted-engine:/var/log/ovirt-engine/engine.log
2021-04-26 16:40:40,260-05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-8) [] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM xxxx.yyyy.zzz command Get
Host Capabilities failed: Internal JSON-RPC error: {'reason':
'internal error: Duplicate key'}
This looks like bad response from vdsm. Can you share the vdsm log from the host
xxxx.yyyy.zzz?
/var/log/vdsm/vdsm.log
Nir
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement:
https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z2GOGEZ2DZX...