Python stack trace for VDSM while monitoring GlusterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17
9:20 a.m.
Hi all,
I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7 GlusterFS
3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on CentOS 7.2):
From /var/log/messages:
Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Extracting Engine VM OVF from the
OVF_STORE
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume path:
/rhev/data-center/mnt/glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c366a972a1
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Found an OVF for HE
VM, trying to convert
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Got vm.conf from
OVF_STORE
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Current state EngineUp
(score: 3400)
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Best remote host
read.mgmt.private (id: 2, score: 3400)
Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:mem_free.MemFree:memFree: 7392
Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:55 shockley ovirt-ha-broker: INFO:cpu_load_no_engine.EngineHealth:System load
total=0.1234, engine=0.0364, non-engine=0.0869
Dec 9 15:27:57 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing VDSM
Dec 9 15:27:57 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting the storage
Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server
Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server
Dec 9 15:27:59 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Refreshing the storage
domain
Dec 9 15:27:59 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod#012
raise convert_to_error(kind, result)#012KeyError: 'device'
From /var/log/vdsm/vdsm.log:
jsonrpc.Executor/1::ERROR::2016-12-09
15:27:46,870::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/5::ERROR::2016-12-09
15:27:48,627::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/7::ERROR::2016-12-09
15:27:50,164::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/0::ERROR::2016-12-09
15:27:52,804::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/5::ERROR::2016-12-09
15:27:54,679::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/2::ERROR::2016-12-09
15:27:58,349::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/4::ERROR::2016-12-09
15:27:59,169::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
From /var/log/vdsm/supervdsm.log:
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
15:27:48,625::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09
15:27:50,163::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09
15:27:52,803::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
15:27:54,677::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09
15:27:58,348::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09
15:27:59,168::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
Please note that the whole oVirt cluster is working (apparently) as it should, but due to
a known limitation with split-GlusterFS-network setup
(http://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in
https://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or present only in
nightly later than 3.6.7, right?) GlusterFS volumes are being managed from the hosts
commandline only, while the oVirt Engine webui is used only to monitor them.
The GlusterFS part is currently experiencing some recurring NFS crashes (using internal
GlusterFS NFS support, not NFS-Ganesha) as reported in Gluster users mailing list and in
Bugzilla (http://www.gluster.org/pipermail/gluster-users/2016-December/029357.html and
https://bugzilla.redhat.com/show_bug.cgi?id=1381970 without any feedback insofar...) but
only on not-oVirt-related volumes.
Finally, I can confirm that checking all oVirt-related and not-oVirt-related GlusterFS
volumes from the hosts commandline with:
vdsClient -s localhost glusterVolumeStatus volumeName=nomevolume
always succeeds without errors.
Many thanks in advance for any advice (please note that I'm planning to upgrade from
3.6.7 to latest nightly 3.6.10.x as soon as the corresponding RHEV gets announced, then
later on all the way up to 4.1.0 as soon as it stabilizes; on GlusterFS-side I'd like
to upgrade asap to 3.8.x but I cannot find any hint on oVirt 3.6.x compatibility...).
Best regards,
Giuseppe
PS: please keep my address in to/copy since I still have problems receiving oVirt mailing
list messages on Hotmail.
2:32 a.m.
On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote:
Hi all,
I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7 GlusterFS
3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on CentOS 7.2):
From /var/log/messages:
Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Extracting Engine VM OVF from the
OVF_STORE
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume path:
/rhev/data-center/mnt/glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c366a972a1
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Found an OVF for HE
VM, trying to convert
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Got vm.conf from
OVF_STORE
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Current state EngineUp
(score: 3400)
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Best remote host
read.mgmt.private (id: 2, score: 3400)
Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:mem_free.MemFree:memFree: 7392
Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:55 shockley ovirt-ha-broker: INFO:cpu_load_no_engine.EngineHealth:System
load total=0.1234, engine=0.0364, non-engine=0.0869
Dec 9 15:27:57 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing VDSM
Dec 9 15:27:57 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting the storage
Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server
Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server
Dec 9 15:27:59 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Refreshing the storage
domain
Dec 9 15:27:59 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection established
Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal server
error#012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest#012 res = method(**params)#012 File
"/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012 result =
fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in
status#012 return self._gluster.volumeStatus(volumeName, brick, statusOption)#012 File
"/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012 rv = func(*args,
**kwargs)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line
50, in __call__#012 return callMethod()#012 File
"/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012 **kwargs)#012
File "<string>", line 2, in glusterVolumeStatus#012 File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'device'
From /var/log/vdsm/vdsm.log:
jsonrpc.Executor/1::ERROR::2016-12-09
15:27:46,870::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/5::ERROR::2016-12-09
15:27:48,627::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/7::ERROR::2016-12-09
15:27:50,164::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/0::ERROR::2016-12-09
15:27:52,804::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/5::ERROR::2016-12-09
15:27:54,679::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/2::ERROR::2016-12-09
15:27:58,349::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/4::ERROR::2016-12-09
15:27:59,169::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
_serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv = func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
**kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
From /var/log/vdsm/supervdsm.log:
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
15:27:48,625::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09
15:27:50,163::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09
15:27:52,803::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
15:27:54,677::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09
15:27:58,348::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09
15:27:59,168::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res = func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in
_parseVolumeStatusDetail
'device': value['device'],
KeyError: 'device'
Please note that the whole oVirt cluster is working (apparently) as it should, but due to
a known limitation with split-GlusterFS-network setup
(http://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in
https://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or present only in
nightly later than 3.6.7, right?) GlusterFS volumes are being managed from the hosts
commandline only, while the oVirt Engine webui is used only to monitor them.
The GlusterFS part is currently experiencing some recurring NFS crashes (using internal
GlusterFS NFS support, not NFS-Ganesha) as reported in Gluster users mailing list and in
Bugzilla (http://www.gluster.org/pipermail/gluster-users/2016-December/029357.html and
https://bugzilla.redhat.com/show_bug.cgi?id=1381970 without any feedback insofar...) but
only on not-oVirt-related volumes.
Finally, I can confirm that checking all oVirt-related and not-oVirt-related GlusterFS
volumes from the hosts commandline with:
vdsClient -s localhost glusterVolumeStatus volumeName=nomevolume
Can you post the output of 'gluster volume status <vol-name> detail --xml'.
Regards,
Ramesh
always succeeds without errors.
Many thanks in advance for any advice (please note that I'm planning to upgrade from
3.6.7 to latest nightly 3.6.10.x as soon as the corresponding RHEV gets announced, then
later on all the way up to 4.1.0 as soon as it stabilizes; on GlusterFS-side I'd like
to upgrade asap to 3.8.x but I cannot find any hint on oVirt 3.6.x compatibility...).
Best regards,
Giuseppe
PS: please keep my address in to/copy since I still have problems receiving oVirt mailing
list messages on Hotmail.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.phx.ovirt.org/mailman/listinfo/users
3:12 p.m.
--_000_HE1PR0901MB112997BB2816E537862FF51BFA9D0HE1PR0901MB1129_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Giuseppe Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare c=
lic sul collegamento seguente.
<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI
[https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...
rv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI
vols.tar.gz<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI
Da: Ramesh Nachimuthu <rnachimu(a)redhat.com Inviato:
luned=EC 12 dicembre 2016 09.32
A: Giuseppe Ragusa
Cc: users(a)ovirt.org
Oggetto: Re: [ovirt-users] Python stack trace for VDSM while monitoring Glu=
sterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17
On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote:
<head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"><!-- P {margin-top:0;margi=
n-bottom:0;} --></style </head <body dir=3D"ltr"
<!--[if lte mso 15 || CheckWebRef]-- <div
id=3D"OwaReferenceAttachments" contenteditable=3D"false" <table style=3D"padding-bottom: 13px; border-width:
0px; border-style: none=
;" <tbody <tr
valign=3D"top" <td <table
style=3D"border-width: 0px 0px 1px 0px; border-color:#C7C7C7; border=
-style: none none dotted none;"
<tbody <tr valign=3D"top" <td style=3D"padding-bottom:7px;" <table align=3D"left" style=3D"padding-right:
28px; border-width: 0px; back=
ground-color: rgb(255, 255, 255); border-spacing: 0px"
<tbody <tr valign=3D"top" <td style=3D"padding: 0px;" <div id=3D"OwaReferenceAttachmentDescription"
style=3D"padding-left: 3px; f=
ont-size: 14px; font-family: 'Segoe UI', 'Segoe WP', 'Segoe UI
WPC', Tahoma=
, Arial, sans-serif; color: rgb(102, 102, 102);" Giuseppe
Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare c=
lic sul collegamento seguente.
</div </td
</tr </tbody </table
</td </tr <tr
valign=3D"top" <td><a
href=3D"https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI" target=3D"_=
blank" <table align=3D"left"
style=3D"padding-right: 28px; padding-bottom:10px; bo=
rder-width: 0px; height:20px; background-color: rgb(255, 255, 255); border-=
spacing: 0px" <tbody <tr valign=3D"top" <td
style=3D"padding: 0px;" <div
style=3D"background-color: rgb(255, 255, 255); height: 20px; width: 20=
px; max-height: 20px;" <a
href=3D"https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI" target=3D"_blan=
k"><img width=3D"20" style=3D"border:0px;"
src=3D"https://r1.res.office365.=
com/owa/prem/images/dc-generic_20.png"></a></div </td <td <div
id=3D"OwaReferenceAttachmentFileName2" style=3D"padding: 0px 0px 0px 5=
px; font-size: 14px; font-family: 'Segoe UI', 'Segoe WP', 'Segoe UI
WPC', T=
ahoma, Arial, sans-serif; color: rgb(0, 114, 198);" <a
href=3D"https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI" target=3D"_blan=
k" style=3D"text-decoration: none; margin: 0px; font-size: 14px; font-famil=
y: 'Segoe UI', 'Segoe WP', 'Segoe UI WPC', Tahoma, Arial,
sans-serif; color=
: rgb(0, 114, 198);">vols.tar.gz</a></div
</td <td
style=3D"display:none;visibility:hidden;" width=3D"0"
height=3D"0"></td=
</tr </tbody </table
</a></td </tr
</tbody </table </td </tr
</tbody </table </div <div
id=3D"OwaReferenceAttachmentsEnd" style=3D"display:none;visibility:hid=
den;"></div <!--[endif]-- <div id=3D"divtagdefaultwrapper"
style=3D"font-size:12pt;color:#000000;font=
-family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr" Da: Ramesh Nachimuthu
&lt;rnachimu(a)redhat.com&gt;<br Inviato:
luned=EC 12 dicembre 2016 09.32<br A:
Giuseppe Ragusa<br Cc: users(a)ovirt.org<br Oggetto: Re: [ovirt-users] Python stack trace for VDSM while
monitoring Glu=
sterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17<br
<br On 12/09/2016 08:50 PM, Giuseppe
Ragusa wrote:<br > Hi all,<br ><br >
I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7=
GlusterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on=
CentOS 7.2):<br ><br > From /var/log/messages:<br ><br >
Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br >
Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine=
_ha.lib.ovf.ovf_store.OVFStore:Extracting Engine VM OVF from the OVF_STORE<=
br > Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine=
_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume path: /rhev/data-center/mnt=
/glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d=
642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-=
a53a-65c366a972a1<br > Dec 9 15:27:47
shockley ovirt-ha-agent: INFO:ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine.config:Found an OVF for HE VM, trying =
to convert<br > Dec 9 15:27:47
shockley ovirt-ha-agent: INFO:ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine.config:Got vm.conf from OVF_STORE<br > Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine:Current state EngineUp (score: 3400)<b=
r > Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine:Best remote host read.mgmt.private (id=
: 2, score: 3400)<br > Dec 9 15:27:48
shockley journal: vdsm jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br >
Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection established<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection closed<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection established<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection closed<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection established<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection closed<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection established<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection closed<br > Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:mem_free.MemFree:m=
emFree: 7392<br > Dec 9 15:27:50
shockley journal: vdsm jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br >
Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br >
Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br >
Dec 9 15:27:55 shockley ovirt-ha-broker: INFO:cpu_load_no_engine=
.EngineHealth:System load total=3D0.1234, engine=3D0.0364, non-engine=3D0.0=
869<br > Dec 9 15:27:57
shockley ovirt-ha-agent: INFO:ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine:Initializing VDSM<br >
Dec 9 15:27:57 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine:Connecting the storage<br > Dec 9 15:27:58 shockley journal: vdsm
jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br >
Dec 9 15:27:58 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine=
_ha.lib.storage_server.StorageServer:Connecting storage server<br > Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine=
_ha.lib.storage_server.StorageServer:Connecting storage server<br > Dec 9 15:27:59 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine=
_ha.lib.storage_server.StorageServer:Refreshing the storage domain<br > Dec 9 15:27:59 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engin=
e_ha.broker.listener.ConnectionHandler:Connection established<br > Dec 9 15:27:59 shockley journal: vdsm
jsonrpc.JsonRpcServer ERRO=
R Internal server error#012Traceback (most recent call last):#012 Fil=
e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line =
533, in _serveRequest#012 res =3D
method(**params)#012&nb=
sp;
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho=
d#012 result =3D fn(*methodArgs)#012 File
"/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs=
p; return self._gluster.volumeStatus(volumeName, brick, statusOption)=
#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in wrapper#012 rv =3D func(*args, **kwargs)#012=
File "/usr/share/vdsm/gluster/api.py", line 407, in volume=
Status#012 statusOption)#012 File
"/usr/share/=
vdsm/supervdsm.py", line 50, in __call__#012 return
=
callMethod()#012 File "/usr/share/vdsm/supervdsm.py",
line 48, in <lambda>#012
**kwargs)#012 File =
"<string>", line 2, in glusterVolumeStatus#012
File &=
quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _=
ca<br >
llmethod#012 raise convert_to_error(kind=
, result)#012KeyError: 'device'<br
><br > From
/var/log/vdsm/vdsm.log:<br ><br > jsonrpc.Executor/1::ERROR::2016-12-09
15:27:46,870::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br >
jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,627::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br >
jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,164::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br >
jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,804::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br >
jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,679::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br >
jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,349::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br >
jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,169::__init__::538::jso=
nrpc.JsonRpcServer::(_serveRequest) Internal server error<br > Traceback (most recent call last):<br > File
"/usr/lib/python2.7/site-packages/yajsonrp=
c/__init__.py", line 533, in _serveRequest<br
> res =3D
method(**params)<br
> File
"/usr/share/vdsm/rpc/Bridge.py", line=
275, in _dynamicMethod<br
> result =3D
fn(*methodArgs)<br
> File
"/usr/share/vdsm/gluster/apiwrapper.py&quo=
t;, line 117, in status<br
> return
self._gluster.volumeStatus(volume=
Name, brick, statusOption)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 86, in wrapper<br
> rv =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/api.py", lin=
e 407, in volumeStatus<br
> statusOption)<br > File
"/usr/share/vdsm/supervdsm.py", line =
50, in __call__<br
> return
callMethod()<br
> File
"/usr/share/vdsm/supervdsm.py", line =
48, in <lambda><br
> **kwargs)<br > File
"<string>", line 2, in glusterV=
olumeStatus<br
> File
"/usr/lib64/python2.7/multiprocessing/mana=
gers.py", line 773, in _callmethod<br
> raise
convert_to_error(kind, result)<br >
KeyError: 'device'<br ><br > From /var/log/vdsm/supervdsm.log:<br ><br >
Traceback (most recent call last):<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br >
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,625::superv=
dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br > Traceback (most recent call last):<br > File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br >
MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,163::superv=
dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br > Traceback (most recent call last):<br > File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br >
MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,803::superv=
dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br > Traceback (most recent call last):<br > File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br >
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,677::superv=
dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br > Traceback (most recent call last):<br > File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br >
MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,348::superv=
dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br > Traceback (most recent call last):<br > File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br >
MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,168::superv=
dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br > Traceback (most recent call last):<br > File
"/usr/share/vdsm/supervdsmServer", li=
ne 118, in wrapper<br
> res =3D func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/supervdsmServer", li=
ne 534, in wrapper<br
> return func(*args,
**kwargs)<br
> File
"/usr/share/vdsm/gluster/cli.py", lin=
e 352, in volumeStatus<br
> return
_parseVolumeStatusDetail(xmltree)=
<br > File
"/usr/share/vdsm/gluster/cli.py", lin=
e 216, in _parseVolumeStatusDetail<br
> 'device':
value['device'],<br > KeyError:
'device'<br ><br > Please note that the whole oVirt cluster is working
(apparently) as it=
should, but due to a known limitation with split-GlusterFS-network setup (=
http://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in ht=
tps://gerrit.ovirt.org/#/c/60083/ but
maybe not backported to 3.6.x or present only in nightly later than 3.6.7,=
right?) GlusterFS volumes are being managed from the hosts commandline onl=
y, while the oVirt Engine webui is used only to monitor them.<br ><br >
The GlusterFS part is currently experiencing some recurring NFS crashe=
s (using internal GlusterFS NFS support, not NFS-Ganesha) as reported in Gl=
uster users mailing list and in Bugzilla (http://www.gluster.org/pipermail/=
gluster-users/2016-December/029357.html
and https://bugzilla.redhat.com/show_bug.cgi?id=3D1381970 without any feed=
back insofar...) but only on not-oVirt-related volumes.<br
><br > Finally, I can confirm that
checking all oVirt-related and not-oVirt-r=
elated GlusterFS volumes from the hosts commandline with:<br ><br >
vdsClient -s localhost glusterVolumeStatus volumeName=3Dnomevolume<br <br Can you post the output of
'gluster volume status <vol-name> detail -=
-xml'.<br <br
Regards,<br Ramesh<br <br Hi Ramesh,<br <br Please find attached all the output
produced with the following command:<br=
<br for vol in $(gluster volume list);
do gluster volume status ${vol} detail -=
-xml > ${vol}.xml; res=3D$?; echo "Exit ${res} for volume ${vol}&qu=
ot;; done<br <br Please
note that the exit code was always zero.<br
<br Many thanks for you help.<br <br Best regards,<br Giuseppe<br
<br <br
><br > always succeeds without
errors.<br ><br > Many thanks in advance for any advice (please note
that I'm planning t=
o upgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the correspondin=
g RHEV gets announced, then later on all the way up to 4.1.0 as soon as it =
stabilizes; on GlusterFS-side I'd like
to upgrade asap to 3.8.x but I cannot find any hint on oVirt 3.6.x compati=
bility...).<br ><br > Best regards,<br >
Giuseppe<br ><br > PS: please keep my address in to/copy since I still
have problems rece=
iving oVirt mailing list messages on Hotmail.<br
><br ><br >
_______________________________________________<br >
Users mailing list<br > Users(a)ovirt.org<br >
http://lists.phx.ovirt.org/mailman/listinfo/users<br
<br <br
</div </body </html
--_000_HE1PR0901MB112997BB2816E537862FF51BFA9D0HE1PR0901MB1129_--
![https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...](https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]%3Chttps://1d=){kind=link}
Hi all,
I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7 Gl=
usterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on Ce=
ntOS 7.2):
From /var/log/messages:
Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern=
al
server error#012Traceback (most recent call last):#012 File "/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.=
ovf.ovf_store.OVFStore:Extracting Engine VM OVF
from the OVF_STORE
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.=
ovf.ovf_store.OVFStore:OVF_STORE volume path:
/rhev/data-center/mnt/gluster=
SD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178=
ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c=
366a972a1
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine.config:Found an OVF
for HE VM, trying to conve=
rt
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine.config:Got vm.conf
from OVF_STORE
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine:Current state
EngineUp (score: 3400)
Dec 9 15:27:47 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine:Best remote host
read.mgmt.private (id: 2, sco=
re: 3400)
Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
Intern=
al server error#012Traceback (most recent call last):#012 File
"/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection
established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection
established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection
established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection
established
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection closed
Dec 9 15:27:48 shockley ovirt-ha-broker:
INFO:mem_free.MemFree:memFree: =
7392
Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
Intern=
al server error#012Traceback (most recent call last):#012 File
"/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
Intern=
al server error#012Traceback (most recent call last):#012 File
"/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
Intern=
al server error#012Traceback (most recent call last):#012 File
"/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
Dec 9 15:27:55 shockley ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineH=
ealth:System load total=3D0.1234, engine=3D0.0364,
non-engine=3D0.0869
Dec 9 15:27:57 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine:Initializing VDSM
Dec 9 15:27:57 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine:Connecting the
storage
Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
Intern=
al server error#012Traceback (most recent call last):#012 File
"/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.=
storage_server.StorageServer:Connecting storage
server
Dec 9 15:27:58 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.=
storage_server.StorageServer:Connecting storage
server
Dec 9 15:27:59 shockley ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.=
storage_server.StorageServer:Refreshing the
storage domain
Dec 9 15:27:59 shockley ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.bro=
ker.listener.ConnectionHandler:Connection
established
Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
Intern=
al server error#012Traceback (most recent call last):#012 File
"/usr/lib/p=
ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0=
12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l=
ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us=
r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self=
._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha=
re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw=
args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus=
#012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in=
__call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p=
y", line 48, in <lambda>#012 **kwargs)#012 File "<string>",
line 2, in =
glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager=
s.py", line 773, in _ca
llmethod#012 raise convert_to_error(kind, result)#012KeyError:
'devi=
ce'
From /var/log/vdsm/vdsm.log:
jsonrpc.Executor/1::ERROR::2016-12-09 15:27:46,870::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,627::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,164::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,804::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,679::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,349::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,169::__init__::538::jsonrp=
c.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53=
3, in _serveRequest
res =3D method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
result =3D fn(*methodArgs)
File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
return self._gluster.volumeStatus(volumeName, brick, statusOption)
File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
rv =3D func(*args, **kwargs)
File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
statusOption)
File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
return callMethod()
File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda **kwargs)
File "<string>", line 2, in glusterVolumeStatus
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in =
_callmethod
raise convert_to_error(kind, result)
KeyError: 'device'
From /var/log/vdsm/supervdsm.log:
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,625::supervdsm=
Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,163::supervdsm=
Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,803::supervdsm=
Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,677::supervdsm=
Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,348::supervdsm=
Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,168::supervdsm=
Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
res =3D func(*args, **kwargs)
File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
return func(*args, **kwargs)
File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
return _parseVolumeStatusDetail(xmltree)
File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus=
Detail
'device': value['device'],
KeyError: 'device'
Please note that the whole oVirt cluster is working (apparently) as it sh=
ould,
but due to a known limitation with split-GlusterFS-network setup (htt=
p://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in https=
://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or present=
only in nightly later than 3.6.7, right?) GlusterFS volumes are being mana=
ged from the hosts commandline only, while the oVirt Engine webui is used o=
nly to monitor them.
The GlusterFS part is currently experiencing some recurring NFS crashes (=
using
internal GlusterFS NFS support, not NFS-Ganesha) as reported in Glust=
er users mailing list and in Bugzilla (http://www.gluster.org/pipermail/glu=
ster-users/2016-December/029357.html and https://bugzilla.redhat.com/show_b=
ug.cgi?id=3D1381970 without any feedback insofar...) but only on not-oVirt-=
related volumes.
Finally, I can confirm that checking all oVirt-related and not-oVirt-rela=
ted
GlusterFS volumes from the hosts commandline with:
vdsClient -s localhost glusterVolumeStatus volumeName=3Dnomevolume
Can you post the output of 'gluster volume status <vol-name> detail --xml'.
Regards,
Ramesh
Hi Ramesh,
Please find attached all the output produced with the following command:
for vol in $(gluster volume list); do gluster volume status ${vol} detail -=
-xml > ${vol}.xml; res=3D$?; echo "Exit ${res} for volume ${vol}"; done
Please note that the exit code was always zero.
Many thanks for you help.
Best regards,
Giuseppe
always succeeds without errors.
Many thanks in advance for any advice (please note that I'm planning to u=
pgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the corresponding R=
HEV gets announced, then later on all the way up to 4.1.0 as soon as it sta=
bilizes; on GlusterFS-side I'd like to upgrade asap to 3.8.x but I cannot f=
ind any hint on oVirt 3.6.x compatibility...).
Best regards,
Giuseppe
PS: please keep my address in to/copy since I still have problems receivi=
ng oVirt
mailing list messages on Hotmail.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.phx.ovirt.org/mailman/listinfo/users
--_000_HE1PR0901MB112997BB2816E537862FF51BFA9D0HE1PR0901MB1129_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html
10:44 p.m.
----- Original Message -----
From: "Giuseppe Ragusa"
<giuseppe.ragusa(a)hotmail.com>
To: "Ramesh Nachimuthu" <rnachimu(a)redhat.com>
Cc: users(a)ovirt.org
Sent: Friday, December 16, 2016 2:42:18 AM
Subject: Re: [ovirt-users] Python stack trace for VDSM while monitoring GlusterFS volumes
in HC HE oVirt 3.6.7 /
GlusterFS 3.7.17
Giuseppe Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare
clic sul collegamento seguente.
<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI>
[https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...
vols.tar.gz<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI>
Da: Ramesh Nachimuthu <rnachimu(a)redhat.com>
Inviato: lunedì 12 dicembre 2016 09.32
A: Giuseppe Ragusa
Cc: users(a)ovirt.org
Oggetto: Re: [ovirt-users] Python stack trace for VDSM while monitoring
GlusterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17
On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote:
> Hi all,
>
> I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7
> GlusterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on
> CentOS 7.2):
>
> From /var/log/messages:
>
> Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Extracting Engine
> VM OVF from the OVF_STORE
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume
> path:
>
/rhev/data-center/mnt/glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c366a972a1
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Found
> an OVF for HE VM, trying to convert
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Got
> vm.conf from OVF_STORE
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Current state
> EngineUp (score: 3400)
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Best remote
> host read.mgmt.private (id: 2, score: 3400)
> Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
> 7392
> Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:55 shockley ovirt-ha-broker:
> INFO:cpu_load_no_engine.EngineHealth:System load total=0.1234,
> engine=0.0364, non-engine=0.0869
> Dec 9 15:27:57 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing
> VDSM
> Dec 9 15:27:57 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting
> the storage
> Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:58 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
> storage server
> Dec 9 15:27:58 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
> storage server
> Dec 9 15:27:59 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Refreshing
> the storage domain
> Dec 9 15:27:59 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
>
> From /var/log/vdsm/vdsm.log:
>
> jsonrpc.Executor/1::ERROR::2016-12-09
> 15:27:46,870::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:48,627::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/7::ERROR::2016-12-09
> 15:27:50,164::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/0::ERROR::2016-12-09
> 15:27:52,804::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:54,679::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/2::ERROR::2016-12-09
> 15:27:58,349::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/4::ERROR::2016-12-09
> 15:27:59,169::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
>
> From /var/log/vdsm/supervdsm.log:
>
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:48,625::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09
> 15:27:50,163::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09
> 15:27:52,803::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:54,677::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09
> 15:27:58,348::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09
> 15:27:59,168::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
>
> Please note that the whole oVirt cluster is working (apparently) as it
> should, but due to a known limitation with split-GlusterFS-network setup
> (http://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in
> https://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or
> present only in nightly later than 3.6.7, right?) GlusterFS volumes are
> being managed from the hosts commandline only, while the oVirt Engine
> webui is used only to monitor them.
>
> The GlusterFS part is currently experiencing some recurring NFS crashes
> (using internal GlusterFS NFS support, not NFS-Ganesha) as reported in
> Gluster users mailing list and in Bugzilla
> (http://www.gluster.org/pipermail/gluster-users/2016-December/029357.html
> and https://bugzilla.redhat.com/show_bug.cgi?id=1381970 without any
> feedback insofar...) but only on not-oVirt-related volumes.
>
> Finally, I can confirm that checking all oVirt-related and
> not-oVirt-related GlusterFS volumes from the hosts commandline with:
>
> vdsClient -s localhost glusterVolumeStatus volumeName=nomevolume
Can you post the output of 'gluster volume status <vol-name> detail
--xml'.
Regards,
Ramesh
Hi Ramesh,
Please find attached all the output produced with the following command:
for vol in $(gluster volume list); do gluster volume status ${vol} detail
--xml > ${vol}.xml; res=$?; echo "Exit ${res} for volume ${vol}"; done
Please note that the exit code was always zero.
+gluster-users
This seems to be a bug in Glusterfs 3.7.17. Output of 'gluster volume status
<vol-name> details --xml ' should have a <device> element for all the
bricks in the volume. But it missing for the arbiter brick. This issue is not
re-producible in Gulsterfs-3.8.
Regards,
Ramesh
![https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...](https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]%3Chttps://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI%3E){kind=link}
Many thanks for you help.
Best regards,
Giuseppe
>
> always succeeds without errors.
>
> Many thanks in advance for any advice (please note that I'm planning to
> upgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the corresponding
> RHEV gets announced, then later on all the way up to 4.1.0 as soon as it
> stabilizes; on GlusterFS-side I'd like to upgrade asap to 3.8.x but I
> cannot find any hint on oVirt 3.6.x compatibility...).
>
> Best regards,
> Giuseppe
>
> PS: please keep my address in to/copy since I still have problems receiving
> oVirt mailing list messages on Hotmail.
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.phx.ovirt.org/mailman/listinfo/users
2702
days inactive
2709
days old
3 comments
2 participants
participants (2)
-
Giuseppe Ragusa -
Ramesh Nachimuthu