Re: [ovirt-users] Python stack trace for VDSM while monitoring GlusterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17
3:09 a.m.
On Tue, Dec 20, 2016, at 09:16, Ramesh Nachimuthu wrote:
----- Original Message -----
> From: "Giuseppe Ragusa" <giuseppe.ragusa(a)hotmail.com>
> To: "Ramesh Nachimuthu" <rnachimu(a)redhat.com>
> Cc: users(a)ovirt.org, gluster-users(a)gluster.org, "Ravishankar
Narayanankutty" <ranaraya(a)redhat.com>
> Sent: Tuesday, December 20, 2016 4:15:18 AM
> Subject: Re: [ovirt-users] Python stack trace for VDSM while monitoring GlusterFS
volumes in HC HE oVirt 3.6.7 /
> GlusterFS 3.7.17
>
> On Fri, Dec 16, 2016, at 05:44, Ramesh Nachimuthu wrote:
> > ----- Original Message -----
> > > From: "Giuseppe Ragusa" <giuseppe.ragusa(a)hotmail.com>
> > > To: "Ramesh Nachimuthu" <rnachimu(a)redhat.com>
> > > Cc: users(a)ovirt.org
> > > Sent: Friday, December 16, 2016 2:42:18 AM
> > > Subject: Re: [ovirt-users] Python stack trace for VDSM while monitoring
> > > GlusterFS volumes in HC HE oVirt 3.6.7 /
> > > GlusterFS 3.7.17
> > >
> > > Giuseppe Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare
> > > clic sul collegamento seguente.
> > >
> > >
> > > <https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI>
> > >
[https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...
> > >
> > > vols.tar.gz<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI>
> > >
> > >
> > >
> > > Da: Ramesh Nachimuthu <rnachimu(a)redhat.com>
> > > Inviato: lunedì 12 dicembre 2016 09.32
> > > A: Giuseppe Ragusa
> > > Cc: users(a)ovirt.org
> > > Oggetto: Re: [ovirt-users] Python stack trace for VDSM while monitoring
> > > GlusterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17
> > >
> > > On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote:
> > > > Hi all,
> > > >
> > > > I'm writing to ask about the following problem (in a HC HE oVirt
3.6.7
> > > > GlusterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup
all
> > > > on
> > > > CentOS 7.2):
> > > >
> > > > From /var/log/messages:
> > > >
> > > > Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > > Dec 9 15:27:47 shockley ovirt-ha-agent:
> > > > INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Extracting
> > > > Engine
> > > > VM OVF from the OVF_STORE
> > > > Dec 9 15:27:47 shockley ovirt-ha-agent:
> > > > INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE
volume
> > > > path:
> > > >
/rhev/data-center/mnt/glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c366a972a1
> > > > Dec 9 15:27:47 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Found
> > > > an OVF for HE VM, trying to convert
> > > > Dec 9 15:27:47 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Got
> > > > vm.conf from OVF_STORE
> > > > Dec 9 15:27:47 shockley ovirt-ha-agent:
> > > > INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Current
> > > > state
> > > > EngineUp (score: 3400)
> > > > Dec 9 15:27:47 shockley ovirt-ha-agent:
> > > > INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Best
> > > > remote
> > > > host read.mgmt.private (id: 2, score: 3400)
> > > > Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > established
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > closed
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > established
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > closed
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > established
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > closed
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > established
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > closed
> > > > Dec 9 15:27:48 shockley ovirt-ha-broker:
> > > > INFO:mem_free.MemFree:memFree:
> > > > 7392
> > > > Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > > Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > > Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > > Dec 9 15:27:55 shockley ovirt-ha-broker:
> > > > INFO:cpu_load_no_engine.EngineHealth:System load total=0.1234,
> > > > engine=0.0364, non-engine=0.0869
> > > > Dec 9 15:27:57 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing
> > > > VDSM
> > > > Dec 9 15:27:57 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting
> > > > the storage
> > > > Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > > Dec 9 15:27:58 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
> > > > storage server
> > > > Dec 9 15:27:58 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
> > > > storage server
> > > > Dec 9 15:27:59 shockley ovirt-ha-agent:
> > > >
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Refreshing
> > > > the storage domain
> > > > Dec 9 15:27:59 shockley ovirt-ha-broker:
> > > >
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
> > > > established
> > > > Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR
> > > > Internal
> > > > server error#012Traceback (most recent call last):#012 File
> > > > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 533, in
> > > > _serveRequest#012 res = method(**params)#012 File
> > > > "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod#012
> > > > result
> > > > = fn(*methodArgs)#012 File
"/usr/share/vdsm/gluster/apiwrapper.py",
> > > > line
> > > > 117, in status#012 return self._gluster.volumeStatus(volumeName,
> > > > brick,
> > > > statusOption)#012 File "/usr/share/vdsm/gluster/api.py",
line 86, in
> > > > wrapper#012 rv = func(*args, **kwargs)#012 File
> > > > "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus#012
> > > > statusOption)#012 File "/usr/share/vdsm/supervdsm.py",
line 50, in
> > > > __call__#012 return callMethod()#012 File
> > > > "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>#012
> > > > **kwargs)#012
> > > > File "<string>", line 2, in glusterVolumeStatus#012
File
> > > > "/usr/lib64/python2.7/multiprocessing/managers.py", line
773, in _ca
> > > > llmethod#012 raise convert_to_error(kind, result)#012KeyError:
> > > > 'device'
> > > >
> > > > From /var/log/vdsm/vdsm.log:
> > > >
> > > > jsonrpc.Executor/1::ERROR::2016-12-09
> > > > 15:27:46,870::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > > jsonrpc.Executor/5::ERROR::2016-12-09
> > > > 15:27:48,627::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > > jsonrpc.Executor/7::ERROR::2016-12-09
> > > > 15:27:50,164::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > > jsonrpc.Executor/0::ERROR::2016-12-09
> > > > 15:27:52,804::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > > jsonrpc.Executor/5::ERROR::2016-12-09
> > > > 15:27:54,679::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > > jsonrpc.Executor/2::ERROR::2016-12-09
> > > > 15:27:58,349::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > > jsonrpc.Executor/4::ERROR::2016-12-09
> > > > 15:27:59,169::__init__::538::jsonrpc.JsonRpcServer::(_serveRequest)
> > > > Internal server error
> > > > Traceback (most recent call last):
> > > > File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
> > > > 533,
> > > > in _serveRequest
> > > > res = method(**params)
> > > > File "/usr/share/vdsm/rpc/Bridge.py", line 275, in
_dynamicMethod
> > > > result = fn(*methodArgs)
> > > > File "/usr/share/vdsm/gluster/apiwrapper.py", line 117,
in status
> > > > return self._gluster.volumeStatus(volumeName, brick,
statusOption)
> > > > File "/usr/share/vdsm/gluster/api.py", line 86, in
wrapper
> > > > rv = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/api.py", line 407, in
volumeStatus
> > > > statusOption)
> > > > File "/usr/share/vdsm/supervdsm.py", line 50, in
__call__
> > > > return callMethod()
> > > > File "/usr/share/vdsm/supervdsm.py", line 48, in
<lambda>
> > > > **kwargs)
> > > > File "<string>", line 2, in glusterVolumeStatus
> > > > File "/usr/lib64/python2.7/multiprocessing/managers.py",
line 773,
> > > > in
> > > > _callmethod
> > > > raise convert_to_error(kind, result)
> > > > KeyError: 'device'
> > > >
> > > > From /var/log/vdsm/supervdsm.log:
> > > >
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > > MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
> > > >
15:27:48,625::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> > > > Error in wrapper
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > > MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09
> > > >
15:27:50,163::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> > > > Error in wrapper
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > > MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09
> > > >
15:27:52,803::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> > > > Error in wrapper
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > > MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09
> > > >
15:27:54,677::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> > > > Error in wrapper
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > > MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09
> > > >
15:27:58,348::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> > > > Error in wrapper
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > > MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09
> > > >
15:27:59,168::supervdsmServer::120::SuperVdsm.ServerCallback::(wrapper)
> > > > Error in wrapper
> > > > Traceback (most recent call last):
> > > > File "/usr/share/vdsm/supervdsmServer", line 118, in
wrapper
> > > > res = func(*args, **kwargs)
> > > > File "/usr/share/vdsm/supervdsmServer", line 534, in
wrapper
> > > > return func(*args, **kwargs)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 352, in
volumeStatus
> > > > return _parseVolumeStatusDetail(xmltree)
> > > > File "/usr/share/vdsm/gluster/cli.py", line 216, in
> > > > _parseVolumeStatusDetail
> > > > 'device': value['device'],
> > > > KeyError: 'device'
> > > >
> > > > Please note that the whole oVirt cluster is working (apparently) as
it
> > > > should, but due to a known limitation with split-GlusterFS-network
> > > > setup
> > > > (http://lists.ovirt.org/pipermail/users/2016-August/042119.html
solved
> > > > in
> > > > https://gerrit.ovirt.org/#/c/60083/ but maybe not backported to
3.6.x
> > > > or
> > > > present only in nightly later than 3.6.7, right?) GlusterFS volumes
are
> > > > being managed from the hosts commandline only, while the oVirt
Engine
> > > > webui is used only to monitor them.
> > > >
> > > > The GlusterFS part is currently experiencing some recurring NFS
crashes
> > > > (using internal GlusterFS NFS support, not NFS-Ganesha) as reported
in
> > > > Gluster users mailing list and in Bugzilla
> > > >
(http://www.gluster.org/pipermail/gluster-users/2016-December/029357.html
> > > > and https://bugzilla.redhat.com/show_bug.cgi?id=1381970 without any
> > > > feedback insofar...) but only on not-oVirt-related volumes.
> > > >
> > > > Finally, I can confirm that checking all oVirt-related and
> > > > not-oVirt-related GlusterFS volumes from the hosts commandline with:
> > > >
> > > > vdsClient -s localhost glusterVolumeStatus volumeName=nomevolume
> > >
> > > Can you post the output of 'gluster volume status <vol-name>
detail
> > > --xml'.
> > >
> > > Regards,
> > > Ramesh
> > >
> > > Hi Ramesh,
> > >
> > > Please find attached all the output produced with the following command:
> > >
> > > for vol in $(gluster volume list); do gluster volume status ${vol} detail
> > > --xml > ${vol}.xml; res=$?; echo "Exit ${res} for volume
${vol}"; done
> > >
> > > Please note that the exit code was always zero.
> > >
> >
> > +gluster-users
> >
> > This seems to be a bug in Glusterfs 3.7.17. Output of 'gluster volume
> > status <vol-name> details --xml ' should have a <device>
element for all
> > the bricks in the volume. But it missing for the arbiter brick. This issue
> > is not re-producible in Gulsterfs-3.8.
>
> Do I need to open a GlusterFS bug for this on 3.7?
> Looking at the changelog, it does not seem to have been fixed in 3.7.18 nor
> to be among the already known issues.
>
Please open a bug against Glusterfs 3.7.17.
Done:
https://bugzilla.redhat.com/show_bug.cgi?id=1406569
![https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https:...](https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]%3Chttps://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI%3E){kind=link}
> On the oVirt side: is GlusterFS 3.8 compatible with oVirt 3.6.x
(maybe with x
> > 7 ie using nightly snapshots)?
>
You can upgrade to GlusterFS 3.8. It is compatible with oVirt 3.6.
Note: You may have to add the GlusterFS 3.8 repo manually from
https://download.gluster.org/pub/gluster/glusterfs/3.8/LATEST/.
Many thanks for your advice and for your help.
Do you happen to know whether the split-GlusterFS-network limitation in oVirt 3.6.7 has
been fixed (backporting https://gerrit.ovirt.org/#/c/60083/ ) in latest oVirt 3.6.x
(unsupported, I know...) nightly snapshot releases?
Many thanks again.
Regards,
Giuseppe
Regards,
Ramesh
> Many thanks.
>
> Regards,
> Giuseppe
>
> > Regards,
> > Ramesh
> >
> >
> > > Many thanks for you help.
> > >
> > > Best regards,
> > > Giuseppe
> > >
> > >
> > > >
> > > > always succeeds without errors.
> > > >
> > > > Many thanks in advance for any advice (please note that I'm
planning to
> > > > upgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the
> > > > corresponding
> > > > RHEV gets announced, then later on all the way up to 4.1.0 as soon
as
> > > > it
> > > > stabilizes; on GlusterFS-side I'd like to upgrade asap to 3.8.x
but I
> > > > cannot find any hint on oVirt 3.6.x compatibility...).
> > > >
> > > > Best regards,
> > > > Giuseppe
> > > >
> > > > PS: please keep my address in to/copy since I still have problems
> > > > receiving
> > > > oVirt mailing list messages on Hotmail.
> > > >
> > > >
> > > > _______________________________________________
> > > > Users mailing list
> > > > Users(a)ovirt.org
> > > > http://lists.phx.ovirt.org/mailman/listinfo/users
> > >
> > >
> > >
>
2893
days inactive
2893
days old
0 comments
1 participants
participants (1)
-
Giuseppe Ragusa