Hi all,

I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7 Gl= usterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on Ce= ntOS 7.2):

From /var/log/messages:

Dec 9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce' Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.= ovf.ovf_store.OVFStore:Extracting Engine VM OVF from the OVF_STORE Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.= ovf.ovf_store.OVFStore:OVF_STORE volume path: /rhev/data-center/mnt/gluster= SD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d642b3178= ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-a53a-65c= 366a972a1 Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine.config:Found an OVF for HE VM, trying to conve= rt Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine.config:Got vm.conf from OVF_STORE Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine:Current state EngineUp (score: 3400) Dec 9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine:Best remote host read.mgmt.private (id: 2, sco= re: 3400) Dec 9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce' Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection established Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection closed Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection established Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection closed Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection established Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection closed Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection established Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection closed Dec 9 15:27:48 shockley ovirt-ha-broker: INFO:mem_free.MemFree:memFree: = 7392 Dec 9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce' Dec 9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce' Dec 9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce' Dec 9 15:27:55 shockley ovirt-ha-broker: INFO:cpu_load_no_engine.EngineH= ealth:System load total=3D0.1234, engine=3D0.0364, non-engine=3D0.0869 Dec 9 15:27:57 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine:Initializing VDSM Dec 9 15:27:57 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine:Connecting the storage Dec 9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce' Dec 9 15:27:58 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.= storage_server.StorageServer:Connecting storage server Dec 9 15:27:58 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.= storage_server.StorageServer:Connecting storage server Dec 9 15:27:59 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.= storage_server.StorageServer:Refreshing the storage domain Dec 9 15:27:59 shockley ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.bro= ker.listener.ConnectionHandler:Connection established Dec 9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERROR Intern= al server error#012Traceback (most recent call last):#012 File "/usr/lib/p= ython2.7/site-packages/yajsonrpc/__init__.py", line 533, in _serveRequest#0= 12 res =3D method(**params)#012 File "/usr/share/vdsm/rpc/Bridge.py", l= ine 275, in _dynamicMethod#012 result =3D fn(*methodArgs)#012 File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 return self= ._gluster.volumeStatus(volumeName, brick, statusOption)#012 File "/usr/sha= re/vdsm/gluster/api.py", line 86, in wrapper#012 rv =3D func(*args, **kw= args)#012 File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus= #012 statusOption)#012 File "/usr/share/vdsm/supervdsm.py", line 50, in= __call__#012 return callMethod()#012 File "/usr/share/vdsm/supervdsm.p= y", line 48, in <lambda>#012 **kwargs)#012 File "<string>", line 2, in = glusterVolumeStatus#012 File "/usr/lib64/python2.7/multiprocessing/manager= s.py", line 773, in _ca llmethod#012 raise convert_to_error(kind, result)#012KeyError: 'devi= ce'

From /var/log/vdsm/vdsm.log:

jsonrpc.Executor/1::ERROR::2016-12-09 15:27:46,870::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device' jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,627::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device' jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,164::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device' jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,804::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device' jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,679::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device' jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,349::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device' jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,169::__init__::538::jsonrp= c.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 53= 3, in _serveRequest res =3D method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMethod result =3D fn(*methodArgs) File "/usr/share/vdsm/gluster/apiwrapper.py", line 117, in status return self._gluster.volumeStatus(volumeName, brick, statusOption) File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper rv =3D func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 407, in volumeStatus statusOption) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeStatus File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in = _callmethod raise convert_to_error(kind, result) KeyError: 'device'

From /var/log/vdsm/supervdsm.log:

Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device' MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,625::supervdsm= Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device' MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,163::supervdsm= Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device' MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,803::supervdsm= Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device' MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,677::supervdsm= Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device' MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,348::supervdsm= Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device' MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,168::supervdsm= Server::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer", line 118, in wrapper res =3D func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer", line 534, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 352, in volumeStatus return _parseVolumeStatusDetail(xmltree) File "/usr/share/vdsm/gluster/cli.py", line 216, in _parseVolumeStatus= Detail 'device': value['device'], KeyError: 'device'

Please note that the whole oVirt cluster is working (apparently) as it sh= ould, but due to a known limitation with split-GlusterFS-network setup (htt=

--_000_HE1PR0901MB112997BB2816E537862FF51BFA9D0HE1PR0901MB1129_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Giuseppe Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare c= lic sul collegamento seguente. <https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI> [https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<https://1d= rv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI> vols.tar.gz<https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI> Da: Ramesh Nachimuthu <rnachimu@redhat.com> Inviato: luned=EC 12 dicembre 2016 09.32 A: Giuseppe Ragusa Cc: users@ovirt.org Oggetto: Re: [ovirt-users] Python stack trace for VDSM while monitoring Glu= sterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17 On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote: p://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in https= ://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or present= only in nightly later than 3.6.7, right?) GlusterFS volumes are being mana= ged from the hosts commandline only, while the oVirt Engine webui is used o= nly to monitor them.

The GlusterFS part is currently experiencing some recurring NFS crashes (=

using internal GlusterFS NFS support, not NFS-Ganesha) as reported in Glust= er users mailing list and in Bugzilla (http://www.gluster.org/pipermail/glu= ster-users/2016-December/029357.html and https://bugzilla.redhat.com/show_b= ug.cgi?id=3D1381970 without any feedback insofar...) but only on not-oVirt-= related volumes.

Finally, I can confirm that checking all oVirt-related and not-oVirt-rela=

ted GlusterFS volumes from the hosts commandline with:

vdsClient -s localhost glusterVolumeStatus volumeName=3Dnomevolume

Can you post the output of 'gluster volume status <vol-name> detail --xml'. Regards, Ramesh Hi Ramesh, Please find attached all the output produced with the following command: for vol in $(gluster volume list); do gluster volume status ${vol} detail -= -xml > ${vol}.xml; res=3D$?; echo "Exit ${res} for volume ${vol}"; done Please note that the exit code was always zero. Many thanks for you help. Best regards, Giuseppe

always succeeds without errors.

Many thanks in advance for any advice (please note that I'm planning to u=

pgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the corresponding R= HEV gets announced, then later on all the way up to 4.1.0 as soon as it sta= bilizes; on GlusterFS-side I'd like to upgrade asap to 3.8.x but I cannot f= ind any hint on oVirt 3.6.x compatibility...).

Best regards, Giuseppe

PS: please keep my address in to/copy since I still have problems receivi=

ng oVirt mailing list messages on Hotmail.

_______________________________________________ Users mailing list Users@ovirt.org http://lists.phx.ovirt.org/mailman/listinfo/users

--_000_HE1PR0901MB112997BB2816E537862FF51BFA9D0HE1PR0901MB1129_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"><!-- P {margin-top:0;margi= n-bottom:0;} --></style> </head> <body dir=3D"ltr"> <!--[if lte mso 15 || CheckWebRef]--> <div id=3D"OwaReferenceAttachments" contenteditable=3D"false"> <table style=3D"padding-bottom: 13px; border-width: 0px; border-style: none= ;"> <tbody> <tr valign=3D"top"> <td> <table style=3D"border-width: 0px 0px 1px 0px; border-color:#C7C7C7; border= -style: none none dotted none;"> <tbody> <tr valign=3D"top"> <td style=3D"padding-bottom:7px;"> <table align=3D"left" style=3D"padding-right: 28px; border-width: 0px; back= ground-color: rgb(255, 255, 255); border-spacing: 0px"> <tbody> <tr valign=3D"top"> <td style=3D"padding: 0px;"> <div id=3D"OwaReferenceAttachmentDescription" style=3D"padding-left: 3px; f= ont-size: 14px; font-family: 'Segoe UI', 'Segoe WP', 'Segoe UI WPC', Tahoma= , Arial, sans-serif; color: rgb(102, 102, 102);"> Giuseppe Ragusa ha condiviso un file di OneDrive. Per visualizzarlo, fare c= lic sul collegamento seguente. </div> </td> </tr> </tbody> </table> </td> </tr> <tr valign=3D"top"> <td><a href=3D"https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI" target=3D"_= blank"> <table align=3D"left" style=3D"padding-right: 28px; padding-bottom:10px; bo= rder-width: 0px; height:20px; background-color: rgb(255, 255, 255); border-= spacing: 0px"> <tbody> <tr valign=3D"top"> <td style=3D"padding: 0px;"> <div style=3D"background-color: rgb(255, 255, 255); height: 20px; width: 20= px; max-height: 20px;"> <a href=3D"https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI" target=3D"_blan= k"><img width=3D"20" style=3D"border:0px;" src=3D"https://r1.res.office365.= com/owa/prem/images/dc-generic_20.png"></a></div> </td> <td> <div id=3D"OwaReferenceAttachmentFileName2" style=3D"padding: 0px 0px 0px 5= px; font-size: 14px; font-family: 'Segoe UI', 'Segoe WP', 'Segoe UI WPC', T= ahoma, Arial, sans-serif; color: rgb(0, 114, 198);"> <a href=3D"https://1drv.ms/u/s!Am_io8oW4r10bw5KMtEtKgpcRoI" target=3D"_blan= k" style=3D"text-decoration: none; margin: 0px; font-size: 14px; font-famil= y: 'Segoe UI', 'Segoe WP', 'Segoe UI WPC', Tahoma, Arial, sans-serif; color= : rgb(0, 114, 198);">vols.tar.gz</a></div> </td> <td style=3D"display:none;visibility:hidden;" width=3D"0" height=3D"0"></td=

</tr> </tbody> </table> </a></td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </div> <div id=3D"OwaReferenceAttachmentsEnd" style=3D"display:none;visibility:hid= den;"></div> <!--[endif]--> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Arial,Helvetica,sans-serif;" dir=3D"ltr"> Da: Ramesh Nachimuthu <rnachimu@redhat.com><br> Inviato: luned=EC 12 dicembre 2016 09.32<br> A: Giuseppe Ragusa<br> Cc: users@ovirt.org<br> Oggetto: Re: [ovirt-users] Python stack trace for VDSM while monitoring Glu= sterFS volumes in HC HE oVirt 3.6.7 / GlusterFS 3.7.17<br> <br> On 12/09/2016 08:50 PM, Giuseppe Ragusa wrote:<br> > Hi all,<br> ><br> > I'm writing to ask about the following problem (in a HC HE oVirt 3.6.7= GlusterFS 3.7.17 3-hosts-replica-with-arbiter sharded-volumes setup all on= CentOS 7.2):<br> ><br> >  From /var/log/messages:<br> ><br> > Dec  9 15:27:46 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> > Dec  9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.lib.ovf.ovf_store.OVFStore:Extracting Engine VM OVF from the OVF_STORE<= br> > Dec  9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume path: /rhev/data-center/mnt= /glusterSD/shockley.gluster.private:_enginedomain/1d60fd45-507d-4a78-8294-d= 642b3178ea3/images/22a172de-698e-4cc5-bff0-082882fb3347/8738287c-8a25-4a2a-= a53a-65c366a972a1<br> > Dec  9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine.config:Found an OVF for HE VM, trying = to convert<br> > Dec  9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine.config:Got vm.conf from OVF_STORE<br> > Dec  9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine:Current state EngineUp (score: 3400)<b= r> > Dec  9 15:27:47 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine:Best remote host read.mgmt.private (id= : 2, score: 3400)<br> > Dec  9 15:27:48 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection established<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection closed<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection established<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection closed<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection established<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection closed<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection established<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection closed<br> > Dec  9 15:27:48 shockley ovirt-ha-broker: INFO:mem_free.MemFree:m= emFree: 7392<br> > Dec  9 15:27:50 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> > Dec  9 15:27:52 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> > Dec  9 15:27:54 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> > Dec  9 15:27:55 shockley ovirt-ha-broker: INFO:cpu_load_no_engine= .EngineHealth:System load total=3D0.1234, engine=3D0.0364, non-engine=3D0.0= 869<br> > Dec  9 15:27:57 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine:Initializing VDSM<br> > Dec  9 15:27:57 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine:Connecting the storage<br> > Dec  9 15:27:58 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> > Dec  9 15:27:58 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.lib.storage_server.StorageServer:Connecting storage server<br> > Dec  9 15:27:58 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.lib.storage_server.StorageServer:Connecting storage server<br> > Dec  9 15:27:59 shockley ovirt-ha-agent: INFO:ovirt_hosted_engine= _ha.lib.storage_server.StorageServer:Refreshing the storage domain<br> > Dec  9 15:27:59 shockley ovirt-ha-broker: INFO:ovirt_hosted_engin= e_ha.broker.listener.ConnectionHandler:Connection established<br> > Dec  9 15:27:59 shockley journal: vdsm jsonrpc.JsonRpcServer ERRO= R Internal server error#012Traceback (most recent call last):#012  Fil= e "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line = 533, in _serveRequest#012    res =3D method(**params)#012&nb= sp; File "/usr/share/vdsm/rpc/Bridge.py", line 275, in _dynamicMetho= d#012    result =3D fn(*methodArgs)#012  File "/us= r/share/vdsm/gluster/apiwrapper.py", line 117, in status#012 &nbs= p;  return self._gluster.volumeStatus(volumeName, brick, statusOption)= #012  File "/usr/share/vdsm/gluster/api.py", line 86, in wrapper#012    rv =3D func(*args, **kwargs)#012=   File "/usr/share/vdsm/gluster/api.py", line 407, in volume= Status#012    statusOption)#012  File "/usr/share/= vdsm/supervdsm.py", line 50, in __call__#012    return = callMethod()#012  File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda>#012    **kwargs)#012  File = "<string>", line 2, in glusterVolumeStatus#012  File &= quot;/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _= ca<br> >   llmethod#012    raise convert_to_error(kind= , result)#012KeyError: 'device'<br> ><br> >  From /var/log/vdsm/vdsm.log:<br> ><br> > jsonrpc.Executor/1::ERROR::2016-12-09 15:27:46,870::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> > jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,627::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> > jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,164::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> > jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,804::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> > jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,679::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> > jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,349::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> > jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,169::__init__::538::jso= nrpc.JsonRpcServer::(_serveRequest) Internal server error<br> > Traceback (most recent call last):<br> >    File "/usr/lib/python2.7/site-packages/yajsonrp= c/__init__.py", line 533, in _serveRequest<br> >      res =3D method(**params)<br> >    File "/usr/share/vdsm/rpc/Bridge.py", line= 275, in _dynamicMethod<br> >      result =3D fn(*methodArgs)<br> >    File "/usr/share/vdsm/gluster/apiwrapper.py&quo= t;, line 117, in status<br> >      return self._gluster.volumeStatus(volume= Name, brick, statusOption)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 86, in wrapper<br> >      rv =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/api.py", lin= e 407, in volumeStatus<br> >      statusOption)<br> >    File "/usr/share/vdsm/supervdsm.py", line = 50, in __call__<br> >      return callMethod()<br> >    File "/usr/share/vdsm/supervdsm.py", line = 48, in <lambda><br> >      **kwargs)<br> >    File "<string>", line 2, in glusterV= olumeStatus<br> >    File "/usr/lib64/python2.7/multiprocessing/mana= gers.py", line 773, in _callmethod<br> >      raise convert_to_error(kind, result)<br> > KeyError: 'device'<br> ><br> >  From /var/log/vdsm/supervdsm.log:<br> ><br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> > MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:48,625::superv= dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> > MainProcess|jsonrpc.Executor/7::ERROR::2016-12-09 15:27:50,163::superv= dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> > MainProcess|jsonrpc.Executor/0::ERROR::2016-12-09 15:27:52,803::superv= dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> > MainProcess|jsonrpc.Executor/5::ERROR::2016-12-09 15:27:54,677::superv= dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> > MainProcess|jsonrpc.Executor/2::ERROR::2016-12-09 15:27:58,348::superv= dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> > MainProcess|jsonrpc.Executor/4::ERROR::2016-12-09 15:27:59,168::superv= dsmServer::120::SuperVdsm.ServerCallback::(wrapper) Error in wrapper<br> > Traceback (most recent call last):<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 118, in wrapper<br> >      res =3D func(*args, **kwargs)<br> >    File "/usr/share/vdsm/supervdsmServer", li= ne 534, in wrapper<br> >      return func(*args, **kwargs)<br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 352, in volumeStatus<br> >      return _parseVolumeStatusDetail(xmltree)= <br> >    File "/usr/share/vdsm/gluster/cli.py", lin= e 216, in _parseVolumeStatusDetail<br> >      'device': value['device'],<br> > KeyError: 'device'<br> ><br> > Please note that the whole oVirt cluster is working (apparently) as it= should, but due to a known limitation with split-GlusterFS-network setup (= http://lists.ovirt.org/pipermail/users/2016-August/042119.html solved in ht= tps://gerrit.ovirt.org/#/c/60083/ but maybe not backported to 3.6.x or present only in nightly later than 3.6.7,= right?) GlusterFS volumes are being managed from the hosts commandline onl= y, while the oVirt Engine webui is used only to monitor them.<br> ><br> > The GlusterFS part is currently experiencing some recurring NFS crashe= s (using internal GlusterFS NFS support, not NFS-Ganesha) as reported in Gl= uster users mailing list and in Bugzilla (http://www.gluster.org/pipermail/= gluster-users/2016-December/029357.html and https://bugzilla.redhat.com/show_bug.cgi?id=3D1381970 without any feed= back insofar...) but only on not-oVirt-related volumes.<br> ><br> > Finally, I can confirm that checking all oVirt-related and not-oVirt-r= elated GlusterFS volumes from the hosts commandline with:<br> ><br> > vdsClient -s localhost glusterVolumeStatus volumeName=3Dnomevolume<br> <br> Can you post the output of 'gluster volume status <vol-name> detail -= -xml'.<br> <br> Regards,<br> Ramesh<br> <br> Hi Ramesh,<br> <br> Please find attached all the output produced with the following command:<br=

<br> for vol in $(gluster volume list); do gluster volume status ${vol} detail -= -xml > ${vol}.xml; res=3D$?; echo "Exit ${res} for volume ${vol}&qu= ot;; done<br> <br> Please note that the exit code was always zero.<br> <br> Many thanks for you help.<br> <br> Best regards,<br> Giuseppe<br> <br> <br> ><br> > always succeeds without errors.<br> ><br> > Many thanks in advance for any advice (please note that I'm planning t= o upgrade from 3.6.7 to latest nightly 3.6.10.x as soon as the correspondin= g RHEV gets announced, then later on all the way up to 4.1.0 as soon as it = stabilizes; on GlusterFS-side I'd like to upgrade asap to 3.8.x but I cannot find any hint on oVirt 3.6.x compati= bility...).<br> ><br> > Best regards,<br> > Giuseppe<br> ><br> > PS: please keep my address in to/copy since I still have problems rece= iving oVirt mailing list messages on Hotmail.<br> ><br> ><br> > _______________________________________________<br> > Users mailing list<br> > Users@ovirt.org<br> > http://lists.phx.ovirt.org/mailman/listinfo/users<br> <br> <br> </div> </body> </html> --_000_HE1PR0901MB112997BB2816E537862FF51BFA9D0HE1PR0901MB1129_--