7:44 a.m.
----- Original Message -----
From: "Giuseppe Ragusa"
<giuseppe.ragusa(a)hotmail.com>
To: "Ramesh Nachimuthu" <rnachimu(a)redhat.com>
Cc: users(a)ovirt.org
Sent: Friday, December 16, 2016 2:42:18 AM
Subject: Re: [ovirt-users] Python stack trace for VDSM while monitoring GlusterFS volumes
in HC HE oVirt 3.6.7 /
GlusterFS 3.7.17
Giuseppe Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare
clic sul collegamento seguente.
<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI>
[https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...
vols.tar.gz<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI>
Da: Ramesh Nachimuthu <rnachimu(a)redhat.com>
Inviato: lunedì 12 dicembre 2016 09.32
A: Giuseppe Ragusa
Cc: users(a)ovirt.org
Oggetto: Re: [ovirt-users] Python stack trace for VDSM while monitoring
GlusterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17
On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote:
> Hi all,
>
> I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7
> GlusterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on
> CentOS 7.2):
>
> From /var/log/messages:
>
> Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Extracting Engine
> VM OVF from the OVF_STORE
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume
> path:
>
/rhev/data-center/mnt/glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c366a972a1
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Found
> an OVF for HE VM, trying to convert
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Got
> vm.conf from OVF_STORE
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Current state
> EngineUp (score: 3400)
> Dec 9 15:27:47 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Best remote
> host read.mgmt.private (id: 2, score: 3400)
> Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:48 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> closed
> Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
> 7392
> Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:55 shockley ovirt-ha-broker:
> INFO:cpu_load_no_engine.EngineHealth:System load total=0.1234,
> engine=0.0364, non-engine=0.0869
> Dec 9 15:27:57 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing
> VDSM
> Dec 9 15:27:57 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting
> the storage
> Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
> Dec 9 15:27:58 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
> storage server
> Dec 9 15:27:58 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
> storage server
> Dec 9 15:27:59 shockley ovirt-ha-agent:
> INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Refreshing
> the storage domain
> Dec 9 15:27:59 shockley ovirt-ha-broker:
> INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> established
> Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Internal
> server error#012Traceback (most recent call last):#012 File
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 533, in
> _serveRequest#012 res = method(**params)#012 File
> "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod#012
result
> = fn(*methodArgs)#012 File "/usr/share/vdsm/gluster/apiwrapper.py", line
> 117, in status#012 return self._gluster.volumeStatus(volumeName, brick,
> statusOption)#012 File "/usr/share/vdsm/gluster/api.py", line 86, in
> wrapper#012 rv = func(*args, **kwargs)#012 File
> "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus#012
> statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in
> __call__#012 return callMethod()#012 File
> "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012
**kwargs)#012
> File "<string>", line 2, in glusterVolumeStatus#012 File
> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _ca
> llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> 'device'
>
> From /var/log/vdsm/vdsm.log:
>
> jsonrpc.Executor/1::ERROR::2016-12-09
> 15:27:46,870::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:48,627::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/7::ERROR::2016-12-09
> 15:27:50,164::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/0::ERROR::2016-12-09
> 15:27:52,804::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:54,679::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/2::ERROR::2016-12-09
> 15:27:58,349::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
> jsonrpc.Executor/4::ERROR::2016-12-09
> 15:27:59,169::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
533,
> in _serveRequest
> res = method(**params)
> File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status
> return self._gluster.volumeStatus(volumeName, brick, statusOption)
> File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper
> rv = func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus
> statusOption)
> File "/usr/share/vdsm/supervdsm.py", line 50, in __call__
> return callMethod()
> File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>
> **kwargs)
> File "<string>", line 2, in glusterVolumeStatus
> File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
> _callmethod
> raise convert_to_error(kind, result)
> KeyError: 'device'
>
> From /var/log/vdsm/supervdsm.log:
>
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:48,625::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09
> 15:27:50,163::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09
> 15:27:52,803::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
> 15:27:54,677::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09
> 15:27:58,348::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
> MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09
> 15:27:59,168::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> Error in wrapper
> Traceback (most recent call last):
> File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper
> res = func(*args, **kwargs)
> File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper
> return func(*args, **kwargs)
> File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus
> return _parseVolumeStatusDetail(xmltree)
> File "/usr/share/vdsm/gluster/cli.py", line 216, in
> _parseVolumeStatusDetail
> 'device': value['device'],
> KeyError: 'device'
>
> Please note that the whole oVirt cluster is working (apparently) as it
> should, but due to a known limitation with split-GlusterFS-network setup
> (http://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in
> https://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or
> present only in nightly later than 3.6.7, right?) GlusterFS volumes are
> being managed from the hosts commandline only, while the oVirt Engine
> webui is used only to monitor them.
>
> The GlusterFS part is currently experiencing some recurring NFS crashes
> (using internal GlusterFS NFS support, not NFS-Ganesha) as reported in
> Gluster users mailing list and in Bugzilla
> (http://www.gluster.org/pipermail/gluster-users/2016-December/029357.html
> and https://bugzilla.redhat.com/show_bug.cgi?id=1381970 without any
> feedback insofar...) but only on not-oVirt-related volumes.
>
> Finally, I can confirm that checking all oVirt-related and
> not-oVirt-related GlusterFS volumes from the hosts commandline with:
>
> vdsClient -s localhost glusterVolumeStatus volumeName=nomevolume
Can you post the output of 'gluster volume status <vol-name> detail
--xml'.
Regards,
Ramesh
Hi Ramesh,
Please find attached all the output produced with the following command:
for vol in $(gluster volume list); do gluster volume status ${vol} detail
--xml > ${vol}.xml; res=$?; echo "Exit ${res} for volume ${vol}"; done
Please note that the exit code was always zero.
+gluster-users
This seems to be a bug in Glusterfs 3.7.17. Output of 'gluster volume status
<vol-name> details --xml ' should have a <device> element for all the
bricks in the volume. But it missing for the arbiter brick. This issue is not
re-producible in Gulsterfs-3.8.
Regards,
Ramesh
![https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...](https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]%3Chttps://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI%3E){kind=link}
Many thanks for you help.
Best regards,
Giuseppe
>
> always succeeds without errors.
>
> Many thanks in advance for any advice (please note that I'm planning to
> upgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the corresponding
> RHEV gets announced, then later on all the way up to 4.1.0 as soon as it
> stabilizes; on GlusterFS-side I'd like to upgrade asap to 3.8.x but I
> cannot find any hint on oVirt 3.6.x compatibility...).
>
> Best regards,
> Giuseppe
>
> PS: please keep my address in to/copy since I still have problems receiving
> oVirt mailing list messages on Hotmail.
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.phx.ovirt.org/mailman/listinfo/users