Yesterday was testday but not much fun :-( I had a reasonable working setup but for testday I decided to start from scratch and that ended rather soon. Installing and configuring engine was not a problem but I want a setup where I have two gluster hosts and two hosts as vmhosts. I added a second cluster using the webinterface set it to gluster storage and added two minimal installed Fedora 18 hosts where I setup static networking and verified that it worked. Adding the two hosts went OK but adding a Volume gives the following error on engine: 2013-02-01 09:32:39,084 INFO [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] (ajp--127.0.0.1-8702-4) [5ea886d] Running command: CreateGlusterVolumeCommand internal: false. Entities affected : ID: 8720debc-a184-4b61-9fa8-0fdf4d339b9a Type: VdsGroups 2013-02-01 09:32:39,117 INFO [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp--127.0.0.1-8702-4) [5ea886d] START, CreateGlusterVolumeVDSCommand(HostName = st02, HostId = e7b74172-2f95-43cb-83ff-11705ae24265), log id: 4270f4ef 2013-02-01 09:32:39,246 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8702-4) [5ea886d] Weird return value: StatusForXmlRpc [mCode=4106, mMessage=XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> ] 2013-02-01 09:32:39,248 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8702-4) [5ea886d] Weird return value: StatusForXmlRpc [mCode=4106, mMessage=XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> ] 2013-02-01 09:32:39,249 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8702-4) [5ea886d] Failed in CreateGlusterVolumeVDS method 2013-02-01 09:32:39,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp--127.0.0.1-8702-4) [5ea886d] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to CreateGlusterVolumeVDS, error = XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> 2013-02-01 09:32:39,254 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp--127.0.0.1-8702-4) [5ea886d] Command CreateGlusterVolumeVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateGlusterVolumeVDS, error = XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> 2013-02-01 09:32:39,255 INFO [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp--127.0.0.1-8702-4) [5ea886d] FINISH, CreateGlusterVolumeVDSCommand, log id: 4270f4ef 2013-02-01 09:32:39,256 ERROR [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] (ajp--127.0.0.1-8702-4) [5ea886d] Command org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateGlusterVolumeVDS, error = XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> 2013-02-01 09:32:39,268 INFO [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] (ajp--127.0.0.1-8702-4) [5ea886d] Lock freed to object EngineLock [exclusiveLocks= key: 8720debc-a184-4b61-9fa8-0fdf4d339b9a value: GLUSTER , sharedLocks= ] 2013-02-01 09:32:40,902 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (QuartzScheduler_Worker-85) START, GlusterVolumesListVDSCommand(HostName = st02, HostId = e7b74172-2f95-43cb-83ff-11705ae24265), log id: 61cafb32 And on ST01 the following in vdsm.log Thread-644::DEBUG::2013-02-01 10:24:06,378::BindingXMLRPC::913::vds::(wrapper) client [192.168.216.150]::call volumeCreate with ('GlusterData', ['st01.nieuwland.nl:/home/gluster-data', 'st02.nieuwland.nl:/home/gluster-data'], 2, 0, ['TCP']) {} MainProcess|Thread-644::DEBUG::2013-02-01 10:24:06,381::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/sbin/gluster --mode=script volume create GlusterData replica 2 transport TCP st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data --xml' (cwd None) MainProcess|Thread-644::DEBUG::2013-02-01 10:24:06,639::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-644::ERROR::2013-02-01 10:24:06,640::supervdsmServer::80::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer.py", line 78, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer.py", line 352, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 45, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 446, in volumeCreate xmltree = _execGlusterXml(command) File "/usr/share/vdsm/gluster/cli.py", line 89, in _execGlusterXml raise ge.GlusterXmlErrorException(err=out) GlusterXmlErrorException: XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> Thread-644::ERROR::2013-02-01 10:24:06,655::BindingXMLRPC::929::vds::(wrapper) vdsm exception occured Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 32, in wrapper rv = func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 63, in volumeCreate transportList) File "/usr/share/vdsm/supervdsm.py", line 81, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 72, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeCreate File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) GlusterXmlErrorException: XML error error: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <cliOutput><volCreate><count>2</count><bricks> st01.nieuwland.nl:/home/gluster-data st02.nieuwland.nl:/home/gluster-data </bricks><transport>tcp</transport><type>2</type><volname>GlusterData</volname><replica-count>2</replica-count></volCreate></cliOutput> And on ST02 I get this error: MainProcess|Thread-93::DEBUG::2013-02-01 10:24:09,540::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/sbin/gluster --mode=script peer status --xml' (cwd None) MainProcess|Thread-93::DEBUG::2013-02-01 10:24:09,622::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-93::DEBUG::2013-02-01 10:24:09,624::BindingXMLRPC::920::vds::(wrapper) return hostsList with {'status': {'message': 'Done', 'code': 0}, 'hosts': [{'status': 'CONNECTED', 'hostname': '192.168.216.152', 'uuid': '15c7a739-6735-43f5-a1c0-3c7ff3469588'}, {'status': 'CONNECTED', 'hostname': '192.168.216.151', 'uuid': 'd53f4dcb-1116-4fbc-8d5b-e882175264aa'}]} Thread-94::DEBUG::2013-02-01 10:24:09,639::BindingXMLRPC::913::vds::(wrapper) client [192.168.216.150]::call volumesList with () {} MainProcess|Thread-94::DEBUG::2013-02-01 10:24:09,641::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/sbin/gluster --mode=script volume info --xml' (cwd None) MainProcess|Thread-94::DEBUG::2013-02-01 10:24:09,724::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 MainProcess|Thread-94::ERROR::2013-02-01 10:24:09,725::supervdsmServer::80::SuperVdsm.ServerCallback::(wrapper) Error in wrapper Traceback (most recent call last): File "/usr/share/vdsm/supervdsmServer.py", line 78, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/supervdsmServer.py", line 352, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 45, in wrapper return func(*args, **kwargs) File "/usr/share/vdsm/gluster/cli.py", line 431, in volumeInfo raise ge.GlusterXmlErrorException(err=[etree.tostring(xmltree)]) GlusterXmlErrorException: XML error error: <cliOutput><opRet>0</opRet><opErrno>0</opErrno><opErrstr /><volInfo><volumes><volume><name>GlusterData</name><id>e5cf9724-ecb0-41a3-abdf-0ea891e49f92</id><type>2</type><status>0</status><brickCount>2</brickCount><distCount>2</distCount><stripeCount>1</stripeCount><replicaCount>2</replicaCount><transport>0</transport><bricks><brick>st01.nieuwland.nl:/home/gluster-data</brick><brick>st02.nieuwland.nl:/home/gluster-data</brick></bricks><optCount>0</optCount><options /></volume><count>1</count></volumes></volInfo></cliOutput> Thread-94::ERROR::2013-02-01 10:24:09,741::BindingXMLRPC::929::vds::(wrapper) vdsm exception occured Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 32, in wrapper rv = func(*args, **kwargs) File "/usr/share/vdsm/gluster/api.py", line 56, in volumesList return {'volumes': self.svdsmProxy.glusterVolumeInfo(volumeName)} File "/usr/share/vdsm/supervdsm.py", line 81, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 72, in <lambda> **kwargs) File "<string>", line 2, in glusterVolumeInfo File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) GlusterXmlErrorException: XML error I have included a lot more logs in the attached zip. Running gluster volume info shows me on both server the correct data. The directory is created under /home, SELinux was enforcing but permissive gives the same error. [root@st01 ~]# rpm -aq | grep vdsm vdsm-gluster-4.10.3-6.fc18.noarch vdsm-xmlrpc-4.10.3-6.fc18.noarch vdsm-cli-4.10.3-6.fc18.noarch vdsm-python-4.10.3-6.fc18.x86_64 vdsm-4.10.3-6.fc18.x86_64 [root@st01 ~]# rpm -aq | grep gluster glusterfs-fuse-3.3.1-8.fc18.x86_64 vdsm-gluster-4.10.3-6.fc18.noarch glusterfs-3.3.1-8.fc18.x86_64 glusterfs-server-3.3.1-8.fc18.x86_64 libvirt-0.10.2.2-3.fc18.x86_64 + deps same version qemu-kvm-1.2.2-2.fc18.x86_64 + deps same version Anything else I can provide do to fix this problem? Joop -- irc: jvandewege @fosdem ;-) Anything else?